Kata defeated so completely that OGS AI review unable to work
(forever cycle)
Score Estimator gives w+0 before capture, and unable to work after capture
(forever cycle)
Score Estimator gives w+0 before capture, and unable to work after capture
(forever cycle)
4 Likes
I’m genuinely happy that fallingsnow88 has finally achieved this. I’ve monitoring these games against katago for a while and wondering what is the endgoal.
1 Like
@hexahedron , I think it’s possible to create special “safe mode”, that is weaker, but do not has bugs. That able to defend against ANY possible exploit.
When Kata thinks where to place stone on next move, it runs simulations of where it may place stone several moves in the future. It assumes that it will not be crazy several moves in the future. So it only thinks how to defend against opponent.
But, what if instead it will assume that it will become crazy several moves in the future? (it will run simulations of placing very random stones). Then it will not only think how to defend against opponent, it will also think how to defend against crazy future self. If not expensive, It will capture group now, not assume that it will be able later. It will actually create 2 eyes more often.
Craziness of future simulation of own color will be separate parameter from craziness of future simulation of enemy color. If set to medium, it will have strength of top human pro, it will not be able to beat bots, but it will be comfortable for other humans. If set to very high, it will have kyu strength and I guess it will play interesting unlike other ways to decrease strength, because it is also way to decrease bugs.
1 Like
I suspect it’s not that easy. It may patch some cases, but it’s not going to be a general fix.
For example prior to the last several months of training, a solidly connected cyclic group that should be dying would wrongly be evaluated as 100% alive just by virtue of its shape, and no matter how many crazy moves you imagine yourself playing, that group would continue to be solidly connected and cyclic, so would continue to be wrongly judged as safe. So unless you run very expensive simulations 100+ moves all the way to the point where the group is physically captured, you won’t even notice there’s a problem.
And full-game-long simulations might not fix the issue in all cases. If you study what selfplay with old nets did, even when the opponent played at full strength you could find some situations where such a dead group would consistently live, because the opponent, even at high playouts, was also convinced the group was alive, and therefore didn’t bother to try to stop the group from incidentally making more eyes via normal endgame moves. There are lots of positions where if you do try to stop extra eyes, it’s very point-losing endgame, so any simulated opponent won’t try to do so unless the simulation of the opponent already has an inkling that there is an issue with life and death… in which case you’re back to the original problem - needing to recognize the threat accurately in the first place so that the simulation of the opponent punishes you correctly.
Practically, it might be a little tricky to operationalize? Like, “try to have a good position assuming that within a few turns you will start making really random moves” probably does incentivize the bot to make more solid eyes. But also probably it incentivizes “pre-connect against all possible ko threats”, which could also be counterproductive if it wastes time defending against lots of 1-move or 2-move threats that would never have allowed an exploit anyways, while ignoring the other part of the board where the opponent is setting up a position where even if you tenuki 4 times you think you’re still alive, but in fact you aren’t. I’m not sure - maybe it works great, but this does trigger a bit programming-danger-sense as being a case where what you literally tell a computer to do to might easily have very different effects from what you intended, and the clueless piece of software will of course do the former and not the latter.
Programming is often like trying to work with the evil genie that follows the strict letter of your wish but finds a way to misinterpret it in a way you would never have imagined (but a bit better since unlike the evil genie, computers aren’t actively hostile, merely clueless). This feels more likely than usual to be one of those cases.
7 Likes
Unfortunately the real world is full of evil genies.
The reason why the research paper about defeating katago was so popular is not just because of go players. Go is a sandbox for AI, with well-defined rules, and no lives at stake.
When AI are applied to medicine or to self-driving cars, or other everyday-life domains with possible immediate life-and-death consequences, these kinds of faults of AI can be really scary. Imagine an AI for self-driving cars, which drives perfectly under “normal” circumstances, but which crashes the car if someone puts up a billboard with a simple trompe-l’oeil.
These circular groups are a bit like a trompe-l’oeil on the side of the road, and if you can find a good general method to defend KataGo against them, that could be of great interest for AI research in general, not just for go players.
2 Likes
Susceptibility to adversarial input attacks are a fairly widespread phenomenon of deep neural networks across a broad range of modalities. Here is an example of a practical demonstration:
1 Like
AI should be able to distinguish how sure it is. In situation where it less sure, car should drive slower than in in situation where it more sure. Don’t just have 2 modes: (to drive / to stop). It never should be 100% sure. Such AI still may be bad at distinguishing real tunnel and picture of tunnel, but it probably will crush slower into picture of tunnel.
“It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so. “ – Mark Twain
2 Likes
I think a problem is that an AI wouldn’t know about which of its own ‘thoughts’ it should be doubtful, where its weaknesses are. It doesn’t know what it doesn’t know.
The areas where AI are weak can also be unexpected for a lay person. For example, AI can make quite realistic images of humans (better than the average human can), but AI often get the number of fingers wrong. A human doesn’t make the same mistakes as an AI, and vice versa.
I don’t expect it will work when we personify AI ‘thought processes’ as if it they are similar to (conscious) human thought processes and try to come up with ‘solutions’ for AI that might work for human minds.
1 Like
Neural net knows that 1-1 move is more doubtful than 4-4 move and that’s why it doesn’t waste playouts on it.
There should be system that able to compare best candidate on move 2 and best candidate on move 4. If best candidate on move 4 is significantly more doubtful than best candidate on move 2, safe mode should be activated for move 4.
(“How to implement it?” is separate question)
What you are describing is a much simpler and more naive approach to what AIs like KataGo or indeed old Leela did with (poorly named) Monte Carlo Tree Search.
Actually, why not do 1 real simulation until double pass? It’s more expensive than move in katago-micro , but less expensive than move in level 1 supporter OGS AI review. I guess fallingsnow88 would win vs current level 1 supporter too. Real simulation may fix it.
Because the point of katago-micro is to let people see how it plays when run with extremely low and cheap settings. Using more visits would already make it avoid a much larger fraction of the losses to the exploit without any change to the search algorithm but increasing the visits would defeat the point of the bot in the first place.
You’re also going to probably need more than 1 “real” simulation to be useful, because one simulation only lets you get one random dice roll of the result for one move. You don’t have anything to compare it to know whether a different move would have been better or worse.
And yes, of course he or she would win against the current level 1 supporter AI, because the current level 1 AI supporter review is catastrophically worse than katago-micro at resisting the exploit, it would be even easier to beat. I’m not sure you realized the review is using old networks that are almost entirely clueless about the problem. Even if you used 1 billion playouts per move with the supporter review network, katago-micro is already doing better than that with merely 40ish playouts. The improvement from the last few months of training has been enormous.
We’re in a decently brisk phase of improvement now - almost every week with each 1-2 new nets, the winrates in a lost position shift by another several % towards the correct value, the first hints of trouble are detected another move or two earlier in many games even if still not early enough, another couple of old games show 15% policy mass instead of 3% on the correct move on the losing turn, etc. Accumulate these over multiple weeks…
Anyways, talk is cheap 
. If you think you have an algorithm that works against all possible exploits and still remains superhuman, try it yourself and publish your results! KataGo is all open source and you wouldn’t be the first person to hack on the code and do cool new things with it.
6 Likes
With my ideas it may become more weak than Ke Jie and it may require strong computer. But do you expect to fix FULL board circle exploit normal way?
Again, talk is cheap. 
I encourage you to try out your ideas yourself and see how well they work! If you know how to code or are willing to learn, I do really think you could try your methods and the community would benefit from any new experimental bots you develop and from seeing in what interesting ways they play well or fail to do so. You’ll also learn a lot.
You also don’t need to interface with KataGo’s C++ code if you find it intimidatingly complex.
For example, you can instead start with a minimalist Go AI in python, such as this tutorial AI bot from this youtube series, which already implements GTP (go text protocol) in just like 20 lines near the bottom of the file and therefore can already play in a basic way with any existing GUI. You can replace the few hundred lines of AI move generation logic above with your own algorithm. And that algorithm can involve querying KataGo, if you like.
For that, here’s a demo on how to query KataGo from python: query_analysis_engine_example.py. If you configure KataGo with numSearchThreads=1 and maxVisits=1, then you can effectively just query for KataGo’s raw net, and then implement any algorithm on top of it that you like in python, including doing a full game simulation if you like just by sending repeated queries. Or if you want to do full-game playouts using more than 1 visit, you can do that. This is not the most computationally efficient, but it definitely works for experimentation.
6 Likes
you may choose best move normal way and then test it with 1 real simulation,
if result would be much worse than what was calculated normal way,
then if 2nd place move is better than result of real simulation of 1st place move, choose 2nd place move
This is ridiculous. They didn’t beat Katago, they exploited the counting system of the server. Correct counting would let White win. Both Chinese- and Japanese rules are unambiguous about that. The statement that this board can’t be counted is simply not true.
Regarding OGS, I’m pretty sure mass abuse of the counting system (even if that works), would lead to mass reporting and bans. Not to rank inflation.
If that works on KGS, that’s a problem of KGS, not a problem of Katago.
Who would ban this people? Three and a half overworked OGS moderators? Or you personally?
I agree with you that the first paper was preposterous. In fact it was flirting with academic misconduct, trying to create a buzz by using deceptive headlines.
However, the second paper was very serious. It described an actual blindspot in Katago, found by an AI. Katago was defeated repeatedly by the adversary AI, and humans were able to reproduce the strategy used to defeat Katago.
On the game linked above by @stone.defender, the human player not only defeated Katago, but killed every stone of Katago on the board. So, the blindspot is very real.
3 Likes