Testing the Volatility: Summary

[espoojaram]

@joooom

the key to judging a good model (if the goal is to judge one by predictive accuracy) might be how well it stratifies players whose observed win rates are significantly different than 50%.

I think we need to discuss the consequences of this, in terms of testing purposes.

So far, I’ve been assuming the obvious way to perform such a test is something like the idea I came up with for a “virtual timeline”, which is just doing the same calculations that “would happen” if the rating system was being used in real life and the same sequence of matches happened.

(Here's the original full explanation of the idea, just for reference, but you don't need to read it)

@espoojaram

Here’s my reasoning: what are the effects that a rating system has on the future games on the site? I’d argue that the main effects are:

1. it affects the matchmaking, though I’d argue all semi-decent rating systems affect the matchmaking in a very similar way, so the effects of this might be limited;
2. it definitely affects the handicap predicted;
3. it (slightly?) affects the performance of players because of the psychological effect of seeing the rank of the opponent, and being affected by being on an upswing or a downswing of the player’s own rank, and having a preconception about what one’s own rank “should” be.

At least for the moment, I don’t think we should care too much about point 3, even though I’m not sure it’s
irrelevant. I’m not sure point 1 is very relevant when it comes to testing the winrate predictions, but I should probably describe the methodology I’m thinking of, before we could discuss that.

Here’s the methodology:

• Collect as many past games as you can process. For example, start from a random player, collect all of their ranked games, start going through their opponents and collect all of their ranked games, iteratively. At some point you stop and you remove from the sample all of the games where one of the players is a player that you hadn’t selected.

Hopefully now you have enough games to accurately sample the skill progression of most of the players you selected.

• Sort the games in chronological order.
• Now essentially simulate a virtual timeline and just run the rating system of your choice on your players (Maybe start each of them with a provisional rating equal to the OGS rating they had in the game you start with? We could try both ways out of curiosity). Obviously, at any point in the virtual timeline, only ever feed to the rating system data points from the past in the virtual timeline.
• And, uh… see what happens.

I’d argue that with the exception of points 1, 2 and 3 above, this is pretty much exactly the same as if you had actually used this rating system in the real world. In fact, as a sanity check and out of curiosity, you could also run the very rating system that was actually in place and see if running it on a subset makes a difference. This could also be a good test of its solidity.

So does point 1, the matchmaking, matter? I don’t know. On the one hand, matchmaking is already imprecise in the real system by necessity, so it’s imprecise in the real one and it’s imprecise in the virtual one. If the general trend of the rating is the same, I think the difference shouldn’t be too much. Also, many games on OGS are not even automatched, and that definitely makes matchmaking slightly less of a factor.

But I would expect the average rating difference between players to be larger in the virtual system than in the real one. And in that case, the virtual rating system is expected to have stronger predictions than a real one. I genuinely don’t know if this makes for a better or a worse test of the virtual rating’s solidity though

Alright, so point 2, the handicap. In a way, I think this is only slightly worse than the matchmaking, as I imagine the main effect being that the simulated system would less often see handicap games where it thinks the handicap is “right”. But in handicap games, the performances of the players are often affected by the number of handicap stones, so this is definitely more of a problem.

So I’d say a virtual simulation should be expected to be less reliable when it comes to handicap games, although again I can’t wrap my head around whether this means it’s a better test or a worse test. Science is hard.

So, because of the way matches are chosen, most of them are between players with similar rating (using what I’m going to call the “real rating system”), which means most of the predictions in the real rating system are close to 50%; whereas in the “virtual rating system” that’s not necessarily the case.

In my original proposal I anticipated this concept and made these considerations:

@espoojaram

So does [the matchmaking] matter? I don’t know. On the one hand, matchmaking is already imprecise in the real system by necessity, so it’s imprecise in the real one and it’s imprecise in the virtual one. If the general trend of the rating is the same, I think the difference shouldn’t be too much. Also, many games on OGS are not even automatched, and that definitely makes matchmaking slightly less of a factor.

But I would expect the average rating difference between players to be larger in the virtual system than in the real one. And in that case, the virtual rating system is expected to have stronger predictions than a real one. I genuinely don’t know if this makes for a better or a worse test of the virtual rating’s solidity though

So, we’ve now discussed the idea that most evaluation metrics, if not all, seem to punish wrong predictions close to 50% less than wrong predictions far from 50%.

I’m trying to think about this, but I’m unable to reach a conclusion.

Do you think this gives an edge to the real rating system, because its wrong predictions, closer to 50%, are punished less? Or do you think this gives an edge to the virtual rating system, because “it’s easier to correctly predict the result if the players have very different ratings”?

Or does it not matter and whichever system gets the “edge” is just the one making better predictions?

There’s a little intuitive voice in my head saying that the answer actually depends on whether you’re using a metric that evaluates games individually, like binomial deviance, or a metric that evaluates based on percentage of correct predictions, like the WHR or the one I had come up with:

@espoojaram

group up all the games where the rating system predicted a similar winrate (say, 55% for Black), and for each of these groups actually count the percentage of wins and compare it to the prediction

But I have no idea how to reason thoroughly about this.