Hi all, I recently implemented the front-end for score-based AI review, and it got me thinking about what else we could do with KataGo’s data.
I was reading this 2020 rank thread and wondered How accurately can we guess somebody’s rank only by looking at their “loss per move (LpM)”?
Imagine if we could have a LpM-based rank where:
Ratings would not drift due to natural deflation
Ratings would be comparable through time (imagine comparing Lee Sedol against Shusaku)
Ratings comparable across different servers and “official” organizations
Two players could play each other over and over again and still see rating improvements
Immediate caveats I can think of:
Fighting styles will probably have a higher LpM
Players that are ahead may try to simplify the game even at the cost of losing point differential. The reverse is also true.
Nonetheless, I think it could be a really fun data analysis project if we had access to a bunch of KataGo reviews and the corresponding players’ rank.
@anoek@BHydden, any thoughts on making such a dataset available?
Katago-based rank
Ultimately, if we really wanted to create a rating system that could assign you a rank based on how you played and not by your wins and losses. We could probably take the KataGo trained networks, and fine-tune it to predict ranks based on players’ moves. This system would have the same benefits as stated above, but should be immune to the caveats too.
A contributor to katrain has been working on something similar, but mostly going by policy values.
There’s probably something interesting in these threads and repo, including datasets with a few thousand analyzed games.
It’s a hard problem though, you can get a reasonable average, but also pretty big errors. I would not recommend ever basing a ranking system on this. Our goal is to identify whether someone should focus on improving their opening/mid game/end game more.
I would really hate if the whole sites ratings would be determined by opinions of one bot. So instead of playing good moves to win and get into higher ranks, you would have to play like the bot does. Would playing traditional fusekis/josekis from 1900’s automatically make you ddk in katas eyes?
Also good to keep in mind, that losing points can be a winning tactic in some situations after your opponent has done a major screw-up. If you’re up by 40, you can throw 20 points on slow and risk-free yose on purpose, and it doesnt make you any worse player (on the contrarily, losing points for avoiding risks is usually a sign of a strong player)
So that should be pretty easy to obtain through the API if you’d like
https://online-go.com/api/v1/games/:game_id:/ai_reviews for a list of all reviews for a game https://online-go.com/termination-api/game/:game_id/ai_review/:ai_review_id for an individual review
no need for keys or anything, just normal http requests and something to parse a lot of json
Absolutely agree here; I think that would be a misapplication of this cool technology. I think the greater possibility for this sort of thing is for cheating detection, as it is used in chess. Of course, the chess community has developed this concept far better than we have, but that’s why we’re having discussions like this, and why some teams are working on this problem. I think that a time will come in the near future when we can use AI to detect cheating with a high degree of accuracy, as it is done in chess.
