Testing the Volatility: Summary

[espoojaram]

I want to put forth a disclaimer: I’m aware that, even though I probably have more mathematical knowledge than a majority of human beings, I am essentially a layman in the context of this conversation, and trying to participate with my ideas kinda makes me a crackpot.

But I have a few things I’d like to hear your opinions on.

First, just cause it might be useful, here's a partial formalization of the concept of "schmating" I described earlier, which I'll refer to as "true rating" (EDIT: or "percentile rating"?) from now on, plus an observation about "transitivity".

---

1. For any pair a,b of players in the population, define P(a,b,t) as the probability a would win over b in a match at time t (I have no idea if function P is well defined, so assume it is, I guess ).

2. Define the function X as X(a,t) = #(b such that P(a,b,t)>0.5) / #(players);
   in simple terms, X(a,t) is the percentage of players that a is “statistically stronger than” at time t.

3. Call a monotonic transformation of X a “true rating” (obviously the ratings of all players have to be calculated using the same monotonic function).

Observation: the function X does define a transitive order in the set of players at any time:

for all a,b,c,t, if X(a,t)>X(b,t) and X(b,t)>X(c,t), then X(a,t)>X(c,t)

but one thing to note about this system is that X(a,t)>X(b,t) doesn’t imply P(a,b,t)>0.5.

One reason is simply because of random fluctuations in the P function, but the more problematic one is that there’s another kind of transitivity that isn’t true:

for all a,b,c, if P(a,b,t)>0.5 and P(b,c,t)>0.5, then P(a,c,t)>0.5 (this is NOT always true)

We would like that to be true, but it generally isn’t, though reasonably, it would be enough if it was “close to being true”, for example in terms of a strong statistical correlation that I can’t quite think of how to formalize.

Then, an observation. I don't believe @joooom has ever said this explicitly, but I feel they might agree with it: a system assigning a scalar rating to every player is in some sense not a one-dimensional system, it's an n-dimensional system where n is the number of players. (Or k times that, if every player is assigned k parameters)

I feel like this would cast some light on why such systems might be so prone to overfitting, or at least this is the way I understand @joooom’s worries.

For example, when trying to optimize the matchmaking, the bias-variance tradeoff (as I understand it so far) doesn’t just apply to every player individually, it applies to the accuracy of the entire system. In fact, perhaps it doesn’t really apply to any individual player at all.

 
 
 Part 1: Analogy to the uncertainty principle for waves

TL;DR: in order to understand a wave-like function, looking at a point sample (or a narrow interval) it too little info. We need to look at the fluctuation itself to get that info.

This is inspired by this popular video. If I understood it and recall it correctly, one takeaway is that when you have a wave, say as a function of time, the less time you spend observing it, the less you can be sure of its properties (in that case wavelength).

 In our case, I believe that the “true rating” acts as a noisy wave fluctuating around a fairly stable curve over time, and for any attempt to “sample” it, there’s a lot of random noise that causes our measure to be inaccurate, which you could think of as another noisy wave being added to the “true rating”.

 What we want to do is not study the frequency spectrum of the wave, but just obtain enough experimental data to be able to know the shape of the wave as accurately as possible – but intuitively, the analogy rings true:

we need to look at more of the wave if we wish to understand it. Any single sample could be on the lower side of the wave, the upper side, or in the middle.

This brings me to two considerations:

1. Even if what we wanted was just to “see through the noise” caused by experimental errors, intuitively we would need to try to guess where the center of the wave is, smoothing out the noise.

2. Even if we could build a good picture of the wave, we would never know which of the fluctuations we observe are noise caused by the limits of the sampling system, or fluctuations in the actual “true rating”.

 Which leads me, intuitively, to the conclusion that we should not try to follow the high-frequency fluctuations at all, and that an approach where we try to keep the rating estimate stable at our best guess of the current “center of gravitation of the wave” is just more likely to be, on average, a good estimate of the “true rating”.

 
 
 Part 2: Why I’m skeptical of instant ratings

TL;DR: for the above reason, I believe they're bound to have a low signal-to-noise ratio.

To be clear, what I said in the last paragraph is what I had been thinking the whole time, at least since I wrote the “schmating” thing. So if you’re thinking “That’s what I’ve been saying this whole time!”, well, it just means you somehow didn’t realize I already agreed with this

 I feel this might be another application of the bias-variance tradeoff principle (in fact, it seems to be essentially what @joooom has said multiple times): in such a system, we would be sacrificing the hope of ever accurately capturing the fluctuations of the “true rating”, but we would have less of a risk of being oversensitive to the noise caused by sources of experimental error (such as the intransitivity of player’s ability).

•  This is the reason I’m highly skeptical of instant ratings: a rating system that updates at every game, with no memory of the shape of the apparent rating fluctuations, feels to me like it’s doomed to either be oversensitive to the noise, likely leading to pointless swirling around until it gets so stupidly far from the actual “true rating” that it’s forced to move back, or to be too slow to notice any big-picture trends in the rating.

And so in the end this is why I have a feeling that reducing volatility might actually end up improving the overall “goodness” of the system.

Then again, as was just pointed out, different metrics of “goodness” might give dramatically different “evaluations” to the same rating system.

 For example, going back to what stone_defender said, it might be better user experience for the system to catch on quickly to the player being in a slump, to help them lose less games in the moment – even if the only way to have that quick reactivity is for the system to also latch on to random noise?

 

Talking about metrics, I have a thought about the metric used in that paper.

 The way it’s defined, based on “the proportion of games whose most likely winner was predicted correctly”, perfectly explains why those measures are all so close to 50%, since of course every prediction is a probabilistic outcome, and most games are between players close in ratings (meaning the prediction is close to 50%).

The first time they brought up that paper, @meili_yinhua used it as an example of how different rating systems are only competing for slight edge improvements, but that metric is so compressed by its own nature that it’s hard to tell.

In this multi-thread conversation, we discussed evaluation metrics somewhat at length, and that was the very reason I was skeptical about one of the proposed evaluation metrics:

@espoojaram

@Jon_Ko

Maybe Brier score - Wikipedia does the trick.

Ah, the good ol’ mean squared error. Now this is something simple enough that even I can understand, so I can get on board

Although I’m thinking, since the only reasonable way to use it, it seems to me, would be to use 1 and 0 for the observed result, could it be a problem that most of the predicted values are very close to 0.5 (because most games, especially the ones we care about the most, have people playing on a supposedly even field)?

It feels like, for any given game considered, most of the difference between a good rating system and a bad one is squashed in pretty much the flattest part of the parabola (around 0.5^2 = 0.25), so I’m worried that it might be too susceptible to random noise.

 
Well, that’s it for now. I had other interesting thoughts, but I need time to organize them in my head, and this message is already long enough