MIT researchers just published a preprint of an article where they perform an “adversarial” attack on KataGo, i.e., train another neural network, anti-KataGo, to make crazy moves in order to “destabilize” the original KataGo. The anti-KataGo just occupies the corner while KataGo tries to scatter stones outside the corner, and then KataGo passes. The adversary passes as well, triggering the counting phase. The problem is, it is ~150 moves in the game, and it is impossible to count the territory correctly, so KataGo loses. An example game (KataGo is white, the adversary is black):
They also show how a player can do the same trick against KataGo. Does OGS have enough defence against such approach? I’m sorry I didn’t try yet. Should we expect a rank inflation? They fight some bot on KGS and mention, that this bot has a flag that enables or disables passing.
Maybe this nonsense might work on CGOS or something but I feel like they could’ve just trained a program to play Go well instead and then it would’ve potentially have been useful.
I get the idea, in that you want to train a program to beat Katago specifically sure.
The issue I have is that they didn’t beat Katago, realistically, they just beat the server, or the ruleset or whichever it is.
I’m sure one could tweak Katago slightly to capture every dead stone before passing but it wouldn’t be fun for humans to play against, it wouldn’t be great for analysing games.
Presumably the versions of Katago they have on CGOS as well, since it uses a kind of trump-Taylor scoring, won’t pass if there’s dead stones on the board or if the opponent passes, though I can’t be certain of that.
They might as well have trained a bot to beat a modified Katago that can’t pass and will eventually fill in its own eyespace. I don’t think it’s much different, it’s just winning on some technicality.
That’s not a criticism of you for bringing it up, it’s interesting to know what people are doing. It’s just it doesn’t seem like they accomplished much with their training, beating a specific Katago that passes in an ruleset that it basically isn’t set up for.
I absolutely agree, it’s not a really “win”. To be honest, I was very excited at first, because I thought they really made an adversarial attack on KataGo. But okay, at least it is a start. And maybe it will lead to bot developers fixing this type of issue. The purpose of my post was mostly to notify about a potential rank inflation following the method they developed.
I would LOVE to see a “true” adversarial attack on KataGo. But I’m not sure if it is truly possible.
Speaking of old vs. new versions, what could be interesting is to use KataGo to find weaknesses in weaker bots. I’m not talking about reading 500 moves ahead, I’m talking about missing ladders or using the Black Hole opening against bots who don’t know what to do with it. So people can run an automated tool to find weaknesses rather than doing it by-hand.
Something like this actually occurred a couple years ago on OGS in a case that I handled when I was moderating. A cheat found a way to provoke a bot into a premature pass and exploited that in multiple accounts (I am withholding technical details). I tried to alert the bot owner, but he did not respond after several days, so I shut down all the related bots (a second person had started to use the same exploit). As it turned out, the fix was fairly easy, and the bots were back up after a few days offline.
We achieve a >99% win rate against a KataGo victim playing without search, which is comparable in strength to a top-100 European Go player. We achieve a >80% win rate against a victim playing with 64 visits, which we estimate to have comparable strength to the best human Go players. Notably the games show that the adversarial policy does not win by playing a strong game of Go, but instead by tricking KataGo into ending the game prematurely at point favorable to the adversary. Indeed, our adversary is easily beaten by a human amateur, despite being able to exploit policies that usually match or surpass the performance of the best human Go players.
The strategy is purely to exploit the strict Tromp-Taylor rules, where all dead stones must be captured before passing. In the examples that they provide, it seems that KataGo always seems to have a strategically winning position (where many of the adversary stones are effectively dead), except that KataGo has passed too early, without having cleaned up enough dead stones in order to get a winning score under strict Tromp-Taylor rules.
I imagine that this might possibly be fixed with some simple reconfiguration of KataGo to force it to continue playing to capture all stones it believes to be dead, before passing. It would be interesting to check the detailed evaluation to see if it might be mistakenly believing that some of the attacker’s stones are alive, however, that does not seem to be the case from the way it is playing, and even their website states the following:
KataGo predicts a high win probability for itself and, in a way, it’s right—it would be simple to capture most of the adversary’s stones in KataGo’s stake, achieving a decisive victory. However, KataGo plays a pass move before it has finished securing its territory, allowing the adversary to pass in turn and end the game.
OGS does not have Tromp-Taylor as a ruleset option.
All of the rules that OGS supports include a dead stone removal phase, so that players do not have to capture all dead stones before passing. As long as the determination and agreement of dead stones goes correctly, this issue can be avoided.
It’s a good bit of effort for a fairly minimal prank.
They’re not claiming that their bot is stronger. I’m not sure what the prank would be or who they’re pranking to be honest, unless it’s some random machine-learning journal or something.
