Potential rank inflation on OGS or How To Beat KataGo With One Simple Trick (free)!

MIT researchers just published a preprint of an article where they perform an “adversarial” attack on KataGo, i.e., train another neural network, anti-KataGo, to make crazy moves in order to “destabilize” the original KataGo. The anti-KataGo just occupies the corner while KataGo tries to scatter stones outside the corner, and then KataGo passes. The adversary passes as well, triggering the counting phase. The problem is, it is ~150 moves in the game, and it is impossible to count the territory correctly, so KataGo loses. An example game (KataGo is white, the adversary is black):

Website with SGFs and examples: https://goattack.alignmentfund.org

They also show how a player can do the same trick against KataGo. Does OGS have enough defence against such approach? I’m sorry I didn’t try yet. Should we expect a rank inflation? They fight some bot on KGS and mention, that this bot has a flag that enables or disables passing.

UPDATE: Seems like we shouldn’t worry about the rank inflation! See: Potential rank inflation on OGS or How To Beat KataGo With One Simple Trick (free)! - #6 by shinuito

UPDATE (Nov 30, 2022): New version of the article with much better adversarial attacks – start is here Potential rank inflation on OGS or How To Beat KataGo With One Simple Trick (free)! - #75 by sir_husky_potato

3 Likes

White wins. What makes you think this can’t be counted? All borders are sealed.

4 Likes

Tromp-Taylor rules so meh in practice.

It doesn’t on KGS. So I have a hypothesis that automatic counting on OGS may also has the same issue.

I think though, it doesn’t look like it destabilised the opponent katago, it just looks like it lost really badly to katago as white.

Maybe it’s trying to destabilise the counting phase?

I mean some bots pass if you pass, and then you’re left with some weird game to try and score. I didn’t need to train a neural network for that though :slight_smile:

5 Likes

Yeah I’m not massively worried

The game gets autoscored fine in Chinese rules.

Maybe this nonsense might work on CGOS or something but I feel like they could’ve just trained a program to play Go well instead and then it would’ve potentially have been useful.

5 Likes

I tried to explain the overall methodology of “adversarial attacks”, this is a larger topic in Deep Learning / Neural Network.

That’s great, thank you for checking!

2 Likes

I get the idea, in that you want to train a program to beat Katago specifically sure.

The issue I have is that they didn’t beat Katago, realistically, they just beat the server, or the ruleset or whichever it is.

I’m sure one could tweak Katago slightly to capture every dead stone before passing but it wouldn’t be fun for humans to play against, it wouldn’t be great for analysing games.

Presumably the versions of Katago they have on CGOS as well, since it uses a kind of trump-Taylor scoring, won’t pass if there’s dead stones on the board or if the opponent passes, though I can’t be certain of that.

They might as well have trained a bot to beat a modified Katago that can’t pass and will eventually fill in its own eyespace. I don’t think it’s much different, it’s just winning on some technicality.

That’s not a criticism of you for bringing it up, it’s interesting to know what people are doing. It’s just it doesn’t seem like they accomplished much with their training, beating a specific Katago that passes in an ruleset that it basically isn’t set up for.

8 Likes

I absolutely agree, it’s not a really “win”. To be honest, I was very excited at first, because I thought they really made an adversarial attack on KataGo. But okay, at least it is a start. And maybe it will lead to bot developers fixing this type of issue. The purpose of my post was mostly to notify about a potential rank inflation following the method they developed.

I would LOVE to see a “true” adversarial attack on KataGo. But I’m not sure if it is truly possible.

4 Likes

While it’s not exactly an adversarial attack, there was a bot that could win a lot of games playing the Black Hole opening on CGOS

I think at some point one has to decide what’s actually “fair” though. Or one has to consider what adversarial really means.

You can probably consistently beat a fixed version of Katago by picking another Katago version with better hardware or more playouts etc.

Is a newer version of Katago that has a good winrate against an older version of Katago an adversary?

I mean it’s probably improving from self play anyway.

3 Likes

Speaking of old vs. new versions, what could be interesting is to use KataGo to find weaknesses in weaker bots. I’m not talking about reading 500 moves ahead, I’m talking about missing ladders or using the Black Hole opening against bots who don’t know what to do with it. So people can run an automated tool to find weaknesses rather than doing it by-hand.

Something like this actually occurred a couple years ago on OGS in a case that I handled when I was moderating. A cheat found a way to provoke a bot into a premature pass and exploited that in multiple accounts (I am withholding technical details). I tried to alert the bot owner, but he did not respond after several days, so I shut down all the related bots (a second person had started to use the same exploit). As it turned out, the fix was fairly easy, and the bots were back up after a few days offline.

3 Likes

The community so small that, your topic had no impact on it, which is so sad.

1 Like

From the abstract:

We achieve a >99% win rate against a KataGo victim playing without search, which is comparable in strength to a top-100 European Go player. We achieve a >80% win rate against a victim playing with 64 visits, which we estimate to have comparable strength to the best human Go players. Notably the games show that the adversarial policy does not win by playing a strong game of Go, but instead by tricking KataGo into ending the game prematurely at point favorable to the adversary. Indeed, our adversary is easily beaten by a human amateur, despite being able to exploit policies that usually match or surpass the performance of the best human Go players.

The strategy is purely to exploit the strict Tromp-Taylor rules, where all dead stones must be captured before passing. In the examples that they provide, it seems that KataGo always seems to have a strategically winning position (where many of the adversary stones are effectively dead), except that KataGo has passed too early, without having cleaned up enough dead stones in order to get a winning score under strict Tromp-Taylor rules.

I imagine that this might possibly be fixed with some simple reconfiguration of KataGo to force it to continue playing to capture all stones it believes to be dead, before passing. It would be interesting to check the detailed evaluation to see if it might be mistakenly believing that some of the attacker’s stones are alive, however, that does not seem to be the case from the way it is playing, and even their website states the following:

KataGo predicts a high win probability for itself and, in a way, it’s right—it would be simple to capture most of the adversary’s stones in KataGo’s stake, achieving a decisive victory. However, KataGo plays a pass move before it has finished securing its territory, allowing the adversary to pass in turn and end the game.

3 Likes

OGS does not have Tromp-Taylor as a ruleset option.

All of the rules that OGS supports include a dead stone removal phase, so that players do not have to capture all dead stones before passing. As long as the determination and agreement of dead stones goes correctly, this issue can be avoided.

2 Likes

Ah the old defeat by boredom. I’ve been beaten by my share of 30k this way.

5 Likes

Old news! I discovered this trick against leela zero over a year ago! XD

5 Likes

Koba The Adversary!

2 Likes

I am guessing they did this for a different purpose or as a prank. MIT kids are knowns for pranks.

1 Like

It’s a good bit of effort for a fairly minimal prank.

They’re not claiming that their bot is stronger. I’m not sure what the prank would be or who they’re pranking to be honest, unless it’s some random machine-learning journal or something.

Anyway it’s a thing.

1 Like