I went looking for some more discussion and found this. from a commenter who says that they are one of the paper authors:
You might find this discussion of our paper at https://www.reddit.com/r/MachineLearning/comments/yjryrd/com… by the lead author of KataGo interesting. He wasn’t that concerned about the rule set, primary concern was that we evaluate in a low-search regime, which is a fair critique. But he overall agrees with our conclusion that self-play just cannot be relied upon to produce robust policies sufficiently OOD.
from that reddit thread, who says that they are the primary katago author:
KataGo might be near-superhuman when in-distribution, which is very much not the case in these positions. There’s no reason to expect it to be so good when out-of-distribution, indeed if 64 visits is just about the bare minimum to be superhuman when in-distribution, then one would generally expect to need more visits to perform well when going even a little out of distribution, much less massively-out-of-distribution like in the examples in this paper.
could someone explain this in around 10k-level go knowledge and 25k-level machine learning knowledge? this stuff about in distribution verses out of distribution?
is “in distribution” playing go in away that it’s typically played and therefore like the training data? and “out of distribution” playing go like a drunken boxer style that is unlike anything in the training data?
Not just the weird style of play, but the fact that game was played on external platform under weird rules and scoring system.
In the paper they claim “standard Tromp-Taylor ruleset for computer Go” which i find pretty far-reached. Main factor on why that game was scored as blacks win.
Under any more commonly used rulesets those black stones outside the corner would have deemed dead and that entire area would have been scored as whites territory. In the game white was the first to pass after b’s T10, indicating that it wasnt programmed for the odd scoring system used with that game.
You can always win any competition by tweaking the rules on your benefit and not telling anyone about it until the game is over ^____^
I’m probably being a bit too harsh to be honest. Apparently Katago is trained with Tromp-Taylor rules, so it is kind of tricking Katago into passing when it’s in very unfamiliar positions like it was said in
I don’t really know or understand enough of the details of it though, but will probably read along with
Try to play with a “Random move” bot - you will get out of distribution pretty quick. I thnk most people make a lot more blunders when they are facing such unfamiliar shapes. Of course it’s hard to check how optimal my moves are if Katago has the same issue…
Some quite strong players use weird opening moves for the same effect. Interesting games, usually, but next time I’ll call the referee to complain about an adversarial attack .
This Ars Technica article, like some other places, is misleading in that it never gives the context that the exploit only works on a specific ruleset and only works at low playouts (for now!), playouts that most experienced AI users on this forum and in the Go community would never have considered particularly reliable in the first place (less than a few 100s of playouts).
That obviously doesn’t rule out that further improvements might make it work at much higher playouts - I think that would be scientifically pretty interesting to see! I’ve been pretty supportive of the authors in the sense that I think their research itself is reasonable from an ML perspective. I will not be surprised in a few months or something they extend their methods to work at higher playouts, although I would actually hope that they pivot slightly to experiments aimed at broader understanding rather than just chasing a higher number (e.g. what other ruleset or other games entirely are vulnerable, whether you can also successfully attack the obvious ways to code a defense against it, etc). I’m definitely curious to see if there’s any further work.
I admit that I’m not all that satisfied with the way the paper authors have chosen to communicate their results. Firstly that they’ve not taken extra-extra-care to emphasize in public communications and clearly highlight they’re legitimately exploiting an oversight in the net in a particular ruleset, emphasizing that it applies to this particular computer ruleset, rather than any of the rulesets that people normally think of in Go. This has led to a lot of widespread pointless misunderstanding about it where many people think their result isn’t legitimate, when in fact it is legitimate, you just have to be clear about these particular rules!
And conversely on the flip side of overselling things rather than the prior issue of accidentally underselling themselves: not clearly communicating the context that so far it only works with lower amounts of search than most would rely on in practice. KataGo capped to 64, or even 1000 playouts, is not a “world-class AI” to begin with. I also understand why they would downplay this. Indeed maybe in a few months they’ll extend it to work at higher playouts. But generally the part where you’re trying to market yourself and play up your strengths and downplay the caveats and fight for recognition/attention in any field gives me a bad taste - it’s why I initially went into industry in a role where I could much more focus on just doing the engineering, rather than going into academia.
It’s not fully the responsibility of the authors what a given news outlet ends up saying versus cutting, but given that most of the article sources directly on an interview with them, I do hope that they did try help the interviewer understand better and that it was the outlet’s issue rather than theirs.
They essentially trained an AI to be a rule douchebag like Robert Jasiek
I wonder did the authors know about this exploit using their human intelligence and thus choose Tromp Taylor * in the hope the AI would find it through training, or did they try others rules too and fail to win and not mention those?
* which, if I understand correctly, is really a mislabelling inaccuracy on KataGo’s part as it has purposefully departed from true Tromp Taylor with the early passes to increase training efficiency and be friendlier to humans playing it. If the KataGo docs instead called it “modified Tromp Taylor” then this rules lawyering AI would fail.
In a sense, their result clearly isn’t legitimate.
Let me make a comparison. Imagine I have discovered a new method to break walnuts with my teeth. Then I publish a paper with title “Revolutionary new method to cure walnut allergies!” in which I describe my walnut-breaking method. Will you say now that my result is legitimate? I did discover a new walnut-breaking method. But then I tried to pretend I had discovered something else. And from the point of view of this something else, my method is not legitimate at all, because it simply doesn’t achieve the result I claim it does.
I would be surprised. Or rather, I wouldn’t be surprised to hear about it, just like I’m not surprised when I hear someone has supposedly proven Riemann’s conjecture or proven that P=NP.
I often have to read papers in which the authors are overselling themselves. It always disappoints me. But there is a line between “overselling”, “not clearly communicating” and “blatantly lying”. Not a very good line to cross, and a hard one to come back from. They already gained a lot of attention in the media, but their reputation among researchers will take a hit.
Yeah, it is.
They made an outrageous claim to have made a revolutionary discovery, and used simple words to make this claim. Then they added a bunch of explanations whose meaning is basically that the first claim is completely false, but those explanations rely on techno babble about playouts and Tromp-Taylor rules.
And then they would deny responsibility when the journalist only keeps the first claim?
It’s a adversarial machine learning paper, not a Go paper. The point of the research is to successfully conduct an adversarial attack on a super-competent target that you might not expect to be vulnerable (KataGo), and they succeeded. This is a perfectly reasonable and interesting result in machine learning; conducting these sorts of exploitative attacks and figuring out how to make systems more robust to them is important both in theory and in the real world. They’re not making an outrageous claim to have made a revolutionary discovery and being exposed as a scam, they’re announcing a very normal sort of result, and as is often the case journalists are overreacting. (I agree that the researchers could have been even more clear about exactly what they’re doing, but it’s not like they’re not trying.)
Yeah, fair enough. We’re in agreement that a lot of the communication and reporting and marketing of this data point has been not great - certainly some of the more popular reporting has made wider of a splash than it has educated listeners/readers to understand the context and details, as popular reporting often does. It also may not be a surprising result given what we already knew about more practical AI blindspots and issues with deep learning in Go - ladder weaknesses, flying dagger joseki - where AlphaZero seems incapable of learning to be fully accurate on own and that already requires some manual intervention.
Would it not be relatively simple to patch KataGo (without retraining any networks) to avoid this issue entirely? Even under this attack, doesn’t KataGo’s evaluation still correctly judge those scattered stones as dead? So, when specifically using strict Tromp-Taylor rules (without dead stone agreement), could you simply configure KataGo to not pass until all dead stones of the opponent have been removed?
Yep, as far as I understand this is very simple to patch. Does anyone care? I could do it with a few lines in next release.
It could be done as a final filter on move selection as you mentioned. Or we could make it so that shallow searches can naturally solve the problem by adding a few lines for a upper-bound-cap on the maximum raw-net probability on pass (so that no adversarial pattern can ever put so much mass on pass that no other move gets searched) and a lower-bound cap on the probability of responding to a pass with a pass (so that the search always at least once considers passing as a reply to passing and therefore can determine for itself if it wins/loses). This way doesn’t even need to write any code to score the position before choosing moves or look for dead stones, you just slightly change the weights in the search.
The search already does this, it’s just that (presumably, I can check in detail later) maybe with a small chance in an adversarially constructed position the net puts a bit too much chance on pass and a bit too little chance on searching pass as a reply to pass to see it in very few playouts.