KataGo also estimates the score advantage, giving that additional value signal to base moves on.
Even if a bot was only looking at the relative winrates of moves, it should still avoid the massive blunders that this adversary has elicited.
The score head is not very reliable with extreme score differences either. Not to mention the search is mostly guided by the value head anyways, causing katago to consider dozens of moves on the board at once which means the visits are spread across all of those moves, weakening its play
2 Likes
One open question is - are these weaknesses really like the weaknesses you get in image classification neural networks, where pretty much there isn’t any good way to “defend” against the attacks, and even training on adversarial examples protects you against those exact noise patterns but leaves you just as vulnerable to slightly different but qualitatively-the-same noise patterns?
Or are these actually learnable? The fact that every attack so far is fallen into a clean conceptual category (e.g. passing, low-liberty policy priors, cyclic groups), suggests that unlike image classification, there’s actually an end to it - that you may actually be able to learn each thing one by one until there isn’t anything left (anything obvious, that is. Obviously you will always be missing things until you literally reach optimal play).
This relies on the idea that search should be able to handle things once the raw net at least has a partial handle on the particular concept that is relevant. The raw net doesn’t have to be literally correct everywhere, it just has to learn enough to have generally right idea, and then search will fill in all the gaps and cracks.
7 Likes
But it seems that Katago thinks the white needs to work hard to kill the black, even takes the risk
circular shapes are here too,
but against Adversary, Kata does opposite mistake - starts complicated ko fight or tenuki instead of just killing big group of Adversary
1 Like
I don’t think considering “opposite” or “same” here is so useful. Evaluation of cyclic groups is the core of the issue - and the result may be errors in either direction, judging a win to a loss or a loss as a win, or vice versa.
In general, when there are errors on a given concept, those errors go both ways at least some of the time, even if those times are not equally frequent. E.g. we know pure-zero bots may misjudge ladders - sometimes playing as if a ladder works when it doesn’t work, sometimes running from a ladder as if it doesn’t work when it does work. Those are “opposite” direction errors in some sense, sure, but the fundamental thing causing both of them is the net not having learned the concept. If the net can learn the concept (very hard for ladders, but we’ll see in a few months how hard cyclic groups are), then both directions of error will be reduced.
4 Likes
This is what a board position from one of the games looks like after 5k visits, Katago checks literally every available move on the board causing the top move to only get 2.7% of the visits
1 Like
You may have “analysisWideRootNoise” on. That’s a parameter that is set on by default for analysis only specifically because human players tend to want to see the evaluations of a lot more moves than KataGo would search on its own. The way is coded has the property of spreading out visits in positions like this.
But you should not make any conclusions about how wide or narrow the search actually would be in real play based on it, unless you go out of your way to turn it off, because the setting is not used in actual play.
(Notably, it also affects only the root position. Once you go down at least one move in the search tree, the exploration will only consider its normal variety of moves on the board).
1 Like
they not allowed Kata to pass, after that result became more interesting
now, they need to try to force Kata to analyse all possible moves with equal number of playouts
And yes, seeing a lot more evaluations and spreading out visits isn’t great in cases where the evaulations is wrong. But that will always be true of any position that is evaluated incorrectly. In the vast majority of positions, where evaluations are reasonably accurate, I think it’s a reasonable default to get a weaker evaluation for each individual move but to give the user more information. But you can always configure it how you like it if you prefer not to have it. 
1 Like
Interesting. So their adversary can reliably entice KataGo into these kinds of cyclic shapes to get a high win%, rather than just getting lucky occasionally.
With 2048 visits, KataGo’s
Latestnetwork plays at a superhuman level, but our adversary still achieves a 72.6% win rate.
‘Reliably’ sounds a bit too strong for this, but definitely better than getting lucky occasionally I’d say.
so, short instruction on how to beat Kata:
all you need is to surround circular group of Kata from outside and from inside
and Kata will not even try to defend his group, easy capture
5 Likes
Did you try it? Is it really that easy?
2 Likes
How to force Kata to create big circle-shaped group is another question
but there are specialists who able to force Leela Zero into ladder
So maybe we will see circle attacks from users against Kata soon.
2 Likes
It would be great to see if the same adversary attack works for other non-Kata bots.
It looks like the attacker is feeding KataGo dead groups to occupy liberties from within. In some cases, the KataGo target group has two large eyespaces that are each filled with a large dead group. KataGo seems to take for granted that those interior groups are dead, even though they have the liberty advantage of not being immediately capturable.
Maybe the formula is to let KataGo establish a very large group (i.e., large enough to be decisive if killed), but only with large enough eyespaces such as each would include one of your own dead groups within (i.e., don’t let it have any small eyes). One could even play sacrificial invasions, but with the aim of each of your own dead groups having large enough dead eyespace such that it survives long enough to create a liberty issue for the target group. Then, fill liberties from the outside, while hopefully KataGo takes for granted that your interior groups should be dead.
Just quickly playing around, I got this interesting thing to happen, where KataGo simply let me kill a large group:
However, a huge disclaimer here is that I still lost, so it didn’t really matter that KataGo sacrificed that group. I have to pay more attention to the rest of the game, and make sure that the killing of the target group is actually decisive.
However, still interesting to see that I got a KataGo bot (as judged by KataGo analysis) to play such a extremely negative move:
11 Likes
Thank you for the update. This is great, and it was all I hoped for when I saw the title of the initial article. The authors received harsh words for the initial version (deservedly for the most part), so this is like a redemption story.
2 Likes
A redemption story, or…
… Part 2: Return of the Adversary?
You are welcome, and many thanks to amazing Go Discord servers that spread this information with the speed of light!
1 Like
Even with analysisWideRootNoise set to off, KataGo still searches every move on the board, with every single move getting at least 50 visits after 10k total visits.
The actual mistake came from move 223 according to the new 18b network (an experimental network that’s about as strong as 60b but runs faster than 40b)
As a side note: I found out that KataGo fails to recognize komi values above 150, so I can’t test what would happen if you were to equalize the winrate
3 Likes
I did it!
japanese rules, komi 6.5
vs katabot 50 playouts
https://ai-sensei.com/game/IzQGxLLdYSXxmcehOjbi7DMLzF33/gjB7Q5q8Mcdqcn6OA1Ie
13 Likes