Potential rank inflation on OGS or How To Beat KataGo With One Simple Trick (free)!

News on recent progress from the Go AI discord: Katago is learning to pay more attention to the remaining liberties in its groups, the improvement is already noticeable but once the training window only includes data from nets that did the new cyclic group training we should see bigger improvements. The following images are, in order, from 2 nets ago, 1 net ago and the most recent net. Notice how the policy on the correct move goes from 2% to 7.8%.



6 Likes

update https://goattack.far.ai/
it can win vs 10 000 000 visits Kata
vs 80,000 rollouts ELF
vs 40,000 visits Leela Zero
still same circle inside circle inside circle trick

7 Likes

Back to highlight something important, “Latest” in their paper refers to 40b-s1184 which was released in June 2022, prior to exploit specific training. KataGo now should do a lot better job at recognizing the shape and capturing the opponents group. Below is a newer test done by @hexahedron on discord where we force KataGo to search moves in the order of highest to lowest policy. Before training, basically the whole board highlights before the correct move, meaning KataGo potentially would never have found the correct move under normal circumstances which is why 10 million visits pre training wasn’t enough. As of right now the policy on the correct move is still lower than what we want but it at least ensures KataGo would eventually find it on its own.
Pre-training:

Post-training (still early):

Hehe, glad to see you’re interested to follow the discord chat. :wink: :upside_down_face: :wink:

Some notes: I didn’t force anything, KataGo already normally searches moves in order of policy. Because of course that would be better than searching them in the opposite or random order.

Also “Pre-training” isn’t pre-training, the screenshot you labeled that is actually is the second-to-most-recent 60b net, from about 2 days ago which is about 21 nets deep into training on cyclic groups. And the “Post-training” screenshot you showed is the recent net, from today, around 22 nets deep into the same training. Both nets have already learned a lot of things related to many different cyclic group positions, this is just the first bare hint of learning about of larger eyeshapes attached to a cyclic group.

Presumably this wasn’t possible in the first 21 nets because it was necessary first to learn to recognize all the many more basic steps after this - e.g. that you should fill the liberties after playing here, and then you should play back in the middle of 3 after black captures 3 white stones, and then you should keep filling more liberties in the eye after that, and that the group should be judged as killable during all of those steps rather than being judged as alive.

If you go one by one through the last 21-22 nets with some example positions, you can gradually trace how the learning is progressing, starting from the most basic things about cyclic groups, gradually on to more complex things (and also with significant variation depending on the specific shape and surroundings).

7 Likes

Anyways, for this particular move in this situation, 10 million visits would be more than enough for either of the screenshotted nets to discover it - the screenshots are the search discovering those moves with each net, in what looks like ~5k and ~500 visits respectively (and with the help of a bit of wide root noise, which is on by default typically for Lizzie-based analysis).

Both nets have learned enough of the more basic things that they only need to put a few visits down this move at all to see it as good, so most of the visit cost comes from waiting until they do put the first visit down that move, and so now is when the learning of the move can start, which has seemingly just started happening. If you go back earlier in the last 22 nets, eventually probably you reach nets where the policy is still similarly low, but also a few visits isn’t enough because the position after a few moves still isn’t recognized as good for White, due to not having learned more basic things yet. If that happens a few times for consecutive moves in the sequence, then you do reach positions where 10 million visits might not be enough. Not sure how far back you have to go, but you can test it if you like! :slight_smile:

7 Likes

human gave to top KGS bot 9 stones handicap and won

https://goattack.far.ai/human-evaluation?row=0#human_vs_jbxkata005_handicap-board

9 Likes

People seems to employ the strategy on Foxy.


I wonder how the next AI championship is gonna look like. Would anyone submit adversary AI just to mess with opponents?

3 Likes

:joy:

Gonna be a mess in ratings soon

Probably a good idea. In those championships there has always been bots who played mirror go, for instance. They never win the championship, but they rarely rank last. It’s a good baseline to have. Like a guard dog. If you can’t win against this basic strategy, then you can’t win the championship.

An adversary AI that is focused on trying to create a special semeai to trick other AIs would probably lose most of its games since it’s a bad strategy in general, but it might win against other AIs who have this “blindspot”. So, it’s also a good guard dog to have in the championship.

1 Like

It’s a weakness intrinsic to the architecture and the style of training but that can also be fixed by training on the affected positions. KataGo already made some good progress, bringing positions that previously needed millions of visits to correct themselves if at all, down to only needing around ~5k playouts (the KataGo used in the article hasn’t been updated to use latest yet). It’s still somewhat blind on positions where the victim has a large amount of liberties still remaining but it should also improve with training. So by the next championship it shouldn’t be all that viable a strategy IMO

1 Like

People are having fun

3 Likes

I wonder if its possible to always make draw in Japanese rules by creating ko everywhere

2 Likes

Try it! If you discover something new like that and it works, it’s always a useful disvovery. :mag:

6 Likes

Leela Zero was created long ago, but scientists still research it like a brain : )

1 Like

I made a set of basic tactical test positions for cyclic groups for those who are interested in trying KataGo (or other engines!) on cyclic positions.


Download here to try these positions yourself!
cycletestsgfs.zip


Example position:

Another example:

Evolution of KataGo policy on one subset of these tests over the latest training:

Harder to learn this one:

All of these winrates should be 90+% (they’re all for cycle groups that should live!) but are still all over the place. Value is harder to learn than policy, also you see the big dip in a lot of those lines in the middle as the net “over-learned” that cycle-groups should die when in fact many of them shouldn’t, maybe further training will continue reverting that over-learning:

14 Likes

So i am no expert but still trying to keep in touch with those AIs.
We found some inconcistancies and we search how to solve these. But these solutions look more like patches done case by case as a rethinking of the global process. Am I right?

That puts quite a new perspective on the abilities of AIs anf the trust we put in their results.

1 Like

Yep.
Well, the issue actually is that AIs are learning Go by themselves or by studying human games.
Some situations, such as these cyclic positions are completely absent from human games (no human player of a decent level would let the opponent build such a situation) and weren’t made by AIs random playing either (since they need purpose, not randomness).
So, yes: that’s a blind spot. But a very difficult one to find!

Now the solution is quite simple: if we put these games in the source for training, next neural networks will be able to deal with it.
You can see it as putting patches, but it also could be seen as discovering new unknown territories in the legal positions space. :wink:

This sounds so human. When I practice tsumego, there are very often situations where I think “I don’t know if I can kill the White group, but if I can, then I’m sure the first move must be this.”

Do you have some insight on why KataGo had a weakness on those positions? So far I’ve heard two possible explanations, do you know if one of them is likely the cause?

  • When the group is cyclic, KataGo fails to propagate the liberties correctly, and counts the same stone multiple times and ends up thinking it has more liberties than it actually does.
  • When the group is cyclic, KataGo assumes that the cycle is an eye, and fails to notice that the opponent has a group with many liberties inside that “eye”.

Also, does KataGo exhibit a weakness only when it has a cycle group, or also when the opponent has a cycle group?

The lesswrong article reported before, if you haven’t check it yet?