Bot's rank fluctuations

[shinuito]

@calebjdastrup

Is it even approaching a limiting value?

Well we know it’s going to be forcefully capped by the maximum set in the code. Then again it’s not like the player is stopping being volatile - in that example the player is jumping from “playing” as a 5kyu or a 15kyu based on a coin toss.

@calebjdastrup

If it was doing what it is supposed to, I would expect it to take fewer than 1000 games to settle on the correct (possibly high) volatility.

Well that also assumes we know what it’s supposed to do. What if it currently is doing what it’s supposed to do it’s just we don’t like that it’s doing that for reason X?

We could test for instance what happens if two ordinary players at the same rating and deviation play with expected results but one has 0.06 volatility and the other has a high value like 0.1, and then track how or if the volatility settles based on the results.

@calebjdastrup

What if you run similarly long simulations for the other players?

I could do that, but the thing is if you just look at every “normal” OGS player, you know that you don’t get a runaway linearly increasing volatility just by observation. They seem to stay around this 0.06 value.

Overall I’d have to read the glicko papers in detail (and I might need a bit of help or more background, or maybe LLMs can help these days) to see where this number comes from, if it or some nearby value is like a stable/fixed point of the algorithm or something like that.

---

The point I was making to @Jon_Ko was that volatility is not just about the level you play at (or so it seems), it’s about the opponents level also.

You don’t see the volatility rise linearly when the opponents are the same rank as the player was at initially.

The example case shown here was when you have the chance to behave in a “volatile” way, where you play a wide range of opponents, and you lose disproportionally often to lower rated opponents than your current rank, that it seemed to increase the volatility linearly.

---

If you have suggestions as to what is worth spending time trying to simulate, I’m interested in doing it. Like if you have some idea in particular to challenge: volatility should increase when X behaves like this or in situation Y; or I expect that volatility should settle at this high value A for reason B.

Or if you have any ideas where you think there might be an issue in the implementation or the glicko2 algorithm itself.

Am I implying that there’s no typos in the code or algorithm? No definitely not. There was a previous issue in the volatility and that was eventually corrected for example.

@anoek

Thanks to the due diligence and persistence of @flovo, it was discovered that the OGS glicko2 implementation had a notable bug in how we used the volatility parameter. In short, we had a line of code that was phi_star = sqrt(player.phi ** 2 + new_volatility * 2) and should have been phi_star = sqrt(player.phi ** 2 + new_volatility ** 2). The difference is that ** means “raise to the power of”, where as * is just multiply by. So, since new_volatility usually has values somewhere around 0.06, the value we were supposed to be adding was somewhere around 0.0036, and instead we were using somewhere around 0.12, so we were off by a factor of about 30.

There was an example of players rating graphs before and after the change given

@anoek

For fun, here’s some before/after screenshots that highlights the effect of the volatility bug:

Before:

After

---