Proposal: New users choose beginner/intermediate/advanced and drop ranked game restrictions

OGS Development
1

This proposal is two-fold:

  1. Redefine starting ranks for new users
  2. Allow all players to play ranked games with any opponent

These could be done individually, but it seems useful to consider them in the same discussion.

I’ve chatted with @anoek and he’s on board in principle. But, need to get some feedback from others on whether this will be the right end result! (and/or to bikeshed the details)

1. Redefine starting ranks for new users

The existing system gives new users a starting rank of ~12kyu (see details at the bottom). This makes the beginner experience frustrating, since they are typically paired with bots and opponents that are much stronger than they are. (Worst of all, they can’t play ranked games with opponents that have the same strength; see part 2 though.)

The proposal is to give users a choice between three strength bands:

  • Beginner: 25k+–12k (range not shown)
  • Intermediate: 16k–1k (range in hover/tooltip/etc.)
  • Advanced: 4k–9d+ (range in hover/tooltip/etc.)

The system then assigns them an initial rating corresponding to 22k, 12k, or 2k, respectively (more details at bottom), although it’s still shown as “?”.

Here is my reasoning:

  • Most importantly, beginners start out with an appropriate rank for a beginner!
  • Beginner vs. Intermediate vs. Advanced is easy for most players to answer (especially beginners). Players who know their rank can choose intelligently.
  • The bands are overlapping to avoid forcing hard choices… players can choose based on which group they want to be paired with to start.
  • The bands are large because the rating system will adjust fairly quickly anyway.
  • The initial rank for each band is in the bottom half, but within 9 stones of the top end of the band.
  • I consulted with a statistician who confirmed it’s safe to use outside knowledge (user context, other rating systems) to inform the starting rating (and deviation). This addressed the main concern @anoek had previously with proposals in this direction.

(NOT part of this proposal: allowing players to import a rank from another Go system, potentially with a lower deviation. But might be interesting to consider in the future.)

2. Allow all players to play ranked games with any opponent

The existing system disallows ranked games with an opponent whose rank difference is more than 9 stones away. This is particularly frustrating for beginners, who are currently dropped in at ~12kyu but a strength of ~25kyu. Even outside of that, it seems like an arbitrary restriction.

The proposal is to allow all players to play ranked games with any opponent. However, ranked games would continue to have a handicap limit of 9 stones.

(NOT part of this proposal: allow ranked games with handicap larger than 9 stones.)

Summary

Existing system:

  • Rating system: Glicko-2.
  • New users have a starting rating of 1500.
  • The rating-to-rank conversion puts 1500 at ~6kyu, but…
    • If the deviation is “high enough”, OGS depresses the rating by the deviation value for pairing.
    • New users have a rating deviation of 350, which is “high enough”.
    • Thus, new users have an effective rank of ~12kyu (as-if rating of 1150).
  • Players can only play ranked games with opponents that are within 9 ranks.

Proposed system:

  • Rating system: Glicko-2.
  • New users have a starting rating of 743 (beginner), 1096 (intermediate), or 1764 (advanced).
  • The rating-to-rank conversion puts that at 22k, 12k, and 2k.
    • Deviation still starts at 350.
    • (NO LONGER adjust effective rank when deviation is high.)
  • Players can play ranked games with any opponent.
    • Handicaps for ranked games are capped to 9.

EDIT: Updated ratings and range display details in response to feedback below. Clarify that ranks are still provisional.

34 Likes
2
  1. Redefine starting ranks for new users

Works for me. Although I’m not a beginner, I recently re-started playing on OGS and having that choice would have been welcome because for honest, experience players it makes a lot of sense. I like the bands as well. Nice touch on the overlap!

  1. Allow all players to play ranked games with any opponent

Works for me. But I don’t understand why this was initially implemented. Is it an issue with how Glicko-2 ratings work? Or because a larger difference would allow for certain injustices to happen?

2 Likes
3

I completly agree for beginners class which is my main concern.
On high levels, i wouldn’t go over 3k entrance point.

1 Like
4

Glicko-2 has no problem with it. I believe the restriction dates from earlier iterations of the OGS rating system, before it used Glicko-2.

5

Restriction made no sense from the beginning. It was better to make games unranked automatically if rank difference is too high.

3 Likes
6

In our last thread on the topic, @meili_yinhua claimed that the 9 stone rule was implemented to prevent extremely slow rank settling:

And indeed when I simulate someone who only plays well above both their “system” and “true” ranks the descent towards the true rank (in this case it’s set to 25k) is significantly slower:

Screenshot 2023-10-29 at 7.08.27 PM
Screenshot 2023-10-29 at 7.08.27 PM1186×886 115 KB

In order to prevent this type of thing, it might make sense to keep a rank-difference restriction, but make the range a function of deviation (in other words, new players can play anyone, less new players more limited)

1 Like
7

What do you mean by:

someone who only plays well above their [rank]

?

  • They win more often than expected?
  • Or that they play weaker players?
  • What is their win/loss rate in this scenario?
  • Is their deviation low or high? Is it held constant, or being updated across the graph?

(Since I’m not sure what the graph is representing, some of what follows is a bit speculative (sorry!).)

a feature that was implemented precisely to deal with this weakness in Glicko design

Is there a chance this was implemented for weaknesses in the original Glicko rating system, before OGS moved to Glicko-2?

In order to prevent this type of thing, it might make sense to keep a rank-difference restriction, but make the range a function of deviation (in other words, new players can play anyone, less new players more limited)

Who would the new players play, if established players can’t play them?

If the slowdown in rank changes is for established players (with low deviation), I’m not sure I see this as a weakness, or what the harm would be in allowing match-ups. IIUC:

  • An established player needs lots of “surprising” results before their deviation increases and their rank starts changes dramatically.
  • A new player already has high deviation and so “surprising” results affect their rank dramatically and immediately.
8

Oh, sorry, there was more context in the link you posted (thanks), I just needed to scroll around a bit.

These seem like the most relevant quotes to pull in from that thread to answer my questions:

Other beginners’ ranks go down because they take games with weak or similarly ranked players. This user’s rank does not go down quickly because they mostly take challenges against SDK

However, I can’t emphasize enough that the effect of few erroneous wins is tiny compared to the effect of only playing strong players [graph]

IIUC, this is the concern: finding the “right” rank is slow if a player ONLY plays opponents that are very far off from the player’s correct rank (and get the expected result most of the time).

A response to that:

  • This seems natural. If a player loses to higher-rated opponent, the rating system learns little (this was the expected result), so it makes sense the rating doesn’t change (much). A player needs to lose to a lower-rated opponent for the rating system to learn something new. EDIT: And the lower-rated the opponent they lose to, the more the system learns.
  • Part (1) of this proposal means that players start out closer to the correct rank, so auto-match will be putting them with players of the correct rank more often. Thanks to that, it seems unlikely that players will accidentally be ONLY playing opponents of vastly different strengths.
  • I’m not sure I see an argument to disallow players intentionally playing ranked games against opponents of a vastly different rating. The games won’t affect their ratings much, but what’s the harm?
2 Likes
9

There can’t be much harm since we’re already allowing those ranked games with >9 difference in ladders and tournaments.

Seems like the top priority by far should be allowing people to play games, so it’s not worth searching so hard to justify this kind of restriction.

8 Likes
10

One example is that they are 12k, but only play 5k players.

The weakness exists in both

That’s a good point. Maybe deviation is not the right number to tie a rank-difference restriction to. I am trying to think of a filter that would prevent someone from playing 300 “unsurprising” games.

I’m not arguing that it’s unnatural. By definition, the simulation is just showing what the system will do in various circumstances. In this case, the “natural” thing provides a suboptimal user experience.

I agree, and I like the proposal! However, sandbaggers/airbaggers are a real problem, and we should probably consider how they will act in a new rating system.

The harm is to the experience of their opponents. Even if your rank doesn’t change much, it’s annoying to play a game that doesn’t match what you were expecting. I would like to say this isn’t such a big deal because opponents can restrict rank on custom challenges, but here is the complaint about a “slow descent” player: Glicko entry points and new players - #96 by stone.defender. (If you click through, the slow descent is originally attributed to a few faulty resignations, but the simulation shows that rank mismatches are a much bigger factor)

11

by that logic unranked games should’t be allowed

12

Weren’t you the person who made the complaint I linked? What were you proposing happen instead? Sorry for the faulty logic :upside_down_face:

13

restrictions may make sense for new accounts, as help, so people play first, get rank fast, understand interface later.

For those who already know their rank and understand OGS, it would be better to have ability to create custom game which ANYONE can accept without making it unranked. So if close enough player accepts, its ranked. If not, it unranked. Same with automatch. So those who ok with 24k - 5k opponents would get game faster.

3 Likes
14

We also have the ability to call a game ranked but have the ranking system not do anything with the result, like with the 7x7 tournament. If for some reason no effect is preferable to a very small effect.

15

TBH, it’s not clear to me how sandbaggers are affected by any ratings proposal :(. A sandbagger can depress their rank by playing (and losing) a bunch of games to bots. Unlikely beginners, they don’t find the experience demoralizing, and they just need to balance wins against beginners with losses against the bots to keep their ranks depressed.

That’s fair, for users that already exist with the wrong rank.

But, by correcting the starting rank of beginners, future new users won’t hit this slow descent problem, because they’ll start out in roughly the right place. (Ignoring bad actors, who can just do whatever they want with bots to manipulate their ranks.)

As a result, (I think?) slow descent is only a problem for beginners that created accounts before the change.

  • Maybe we just let the problem naturally disappear over time (as the slow descent for these historical accounts take place, eventually the problem will peter out).
  • Or, we could do a one-time rating correction, after the new thing goes live, only adjusting accounts that fit the slow-descent profile.

WDYT?

1 Like
16

Yep just for historical clarification, we never used Glicko v1, we used Elo when we were first getting going. Elo moves a lot slower in general, but at the same time will move too fast when there’s a big rating difference. Without the concept of RD it can be pretty damaging when one of the players has a very incorrect rating.

Anyways, I think all of that is fairly moot now, especially as the volatility issue was fixed a few years back resulting in notably lower RD’s for established players.

6 Likes
17

I might go a bit further: that 9 stones + 2 Ratings Deviations for handi and 2 Ratings Deviations for non-handi in ranked games (with a floor of 200 or so rating points difference)

Unrainked becomes/remains unrestricted
(i forget how it is rn)

18

Ah, so the in-practice frustrations with Glicko-2’s performance were (we think) mostly caused by the implementation bug in volatility?

19

Yep, although it was still a lot better than Elo, bugs n all :slight_smile: But yeah as it relates to the 9 rank limit thing, I wouldn’t go so far as to say it was due to that specifically or anything, more like I would have not wanted to change too much all at once when we first switched over, followed up by a period of time where it probably would still not make sense (because of the increased volatility), and now we’re a few years down the line and it’s more like I just haven’t really given it any thought in awhile. But stopping to ponder now, and playing with some quick numbers about how it’d affect different players, it seems like it should be just fine.

12 Likes
20

Why? Just so people aren’t catapulted in the dan-levels? Or so they’re not bothered too much with spam games? Or something else? (Not disagreeing, just wondering what your specific concern is…)

I had a look. If we pick these entry points:

  • 800 (beginner) — 20kyu
  • 1250 (intermediate) — 10kyu
  • 1700 (advanced) — 2.6kyu 2.8kyu

… then we have a “nice” (but probably irrelevant?) property that the entries are in 450 rating point increments.

Anyway, I’m just as happy with 2.8kyu as I am with ~1kyu as an entry point for advanced. If that’s more palatable for some, then SGTM. (I assume no one has concerns with it?)

EDIT: I misread the result from the ratings calculator at first.

4 Likes