Oh wait, so you can still play ranked games against bots, and it counts towards the bot’s rating, but not towards the human player’s rating? o.O
As anoek said in the announcement of this, samples of games with humans are taken to rank bots.
This sampling occurs to avoid the adverse effect of people playing repeated ranked games against a bot and smashing its rank: it makes sure that the bot’s rank is based on a proper random sample of games.
That said, this looks wrong: it’s as if the sampling is chosing mostly wins!
1 Like
The sampling should also choose even games, not handicap games. For instance this bot’s rank is still way off.
Interesting, it’s not actually random, it’s simply 1 game per day per player max, game must go to scoring, and the human player must have an established rating.
There are lots of bots that don’t have that spike and it appears like it might be flattening out, so I think the bot might have just been really suppressed before.
I wonder whether “game must go to scoring” biases the selection to games where the bot eventually wins?
(Though I can’t explain why this would be the right psychology!)
The pie charts of wins vs losses on each day since this came in have remarkably few losses in them…
I wondered the same thing initially but then it’s the bot who is winning mostly, and that’s not shared with many of the other easy bots, so 
For example, GnuGo is largely unchanged and has a ton of games being played:
The difference is that noob_bot_1 doesn’t play handicap games, contrary to GnuGo.
So maybe its because noob_bot1 tends to resign when its not winning? For example, games like this, where it resigned against 9k when being behind by 3 points 친선 대국
3 Likes
That would make a lot of sense for a sampling bias, if a bot tends to resign when losing rather than go to scoring, you’d expect to see more disproportionately more winning games for the bot at scoring. It could also be a bot specific behaviour, if some bots tend to prefer to go to scoring whether losing or not, and some have a preference for resigning at certain thresholds.
2 Likes
That would do it, I’ll update the system so that if the bot resigns it counts that too (assuming the other criteria are met as well.)
2 Likes
I’m working on a bot right now (5k-kgs-humanlike-kata) and just got exposed to the weird world of bot games.
If I understand correctly, assuming other criteria are met:
- bot resigns → counts
- human resigns → doesn’t count
- goes to scoring → counts
I assume you are trying to prevent people from sandbagging and lowering their ranks, but now there’s a downward pressure on bot ranks. Bots can only rank up by making sure the game lasts until the end.
I’m pretty invested in this topic right now. I’d love to chat/dm directly with the devs to sketch out some analysis.
2 Likes
im gonna make a wild guess and predict that it slowly drops below previous average
noob_bot_1 rank now looks same as before all these updates