When looking through the games I played using the Kyu Supporter AI, it sometimes happens that while playing out the suggested moves, on move 5 or 6 KataGo realizes: " Welp, nevermind don’t listen to me, if you play like this you are actually down 5 points".
Is this to be expected given the complex nature of a Go match, or would this improve significantly with more runouts f.e. 1000 instead of 400?
Did maybe someone here test 400 and 1000 runouts side by side and can tell wether there’s a difference?
(I mostly play 9x9 and this doesn’t happen very often)
If you have a particular game in mind, I can run a level 4 review for you and we can find out.
I’m also interested in the results.
That said I’ve seen a game between two bots both running katago, where one katago must’ve overlooked some variation and when forced to study it, Ogs AI review included, it realises that it’s estimate is off by 5 points. I’ll find the game again for reference. That’s probably a somewhat exceptional case since it’s two superhuman bots playing each other
That sounds good!
I’ve spent the better part of the last hour looking for branches in which the AI is off, just to have all of them disappear.
I’m kind of tilted for now but I’m sure it will happen again and then I’ll know how to save it and will post it here.
As the playouts increase, the evaluation estimates converge and stabilize. More is almost always better. But sometimes it takes tens of thousands of playouts to judge a situation accurately, even for 9x9. See how many evaluation mistakes the level 4 AI made in my recent game below:
That’s wild. For it to realize that it is off by 8 points with move 29 is quite a bit. I suppose it’s hard to wrap ones head around how complex even 9x9 is.
Could you please elaborate more on this?
I’m surprised to see many positive values (moves 12 and 13 are some astonishing +15 and +17% winrate) but also many turns in which all possible moves have negative values (move 7 best option is -4.7%).
That’s weird, isn’t it?
Well, it’s weird only if you consider the site AI as an oracle that anticipates every eventuality. We all know that just isn’t true. But you have to play complex games like this to see the AI’s shortcomings. This game was particularly hard for the AI. It didn’t have enough playouts to see the path it thought led to a +15% swing. It also didn’t have enough playouts to see that its +15% expectation could be refuted, and the AI reneged on its earlier optimism. Sounds familiar!
Maybe it’s just hard to emulate the complexity of 9x9 on 19x19.
What I mean is how often can one start absolutely massive capturing races which are very close in liberties with many tesujis which the ai would struggle to read.
The capturing races on 9x9 are quite large in scale in comparison to the number of points on the board, like a large fraction of the stones on the board life and death can hinge on it.
Here I’ve got a match I just played On move 50 the AI says black is winning by 2,5 points, but if I play out the endgame as suggested by the AI, white wins by 7,5p. Would be sweet so see how a higher level AI handles the match!
Is black possibly not winning by 2,5 points in the first place or does the stronger AI find the correct sequence for black to win?
(10 points over the course of 30 moves doesn’t seem like a huge change, but then again it’s not like there’s any territory to be gained)
Thanks, it’s pretty nice to have the level 1 and level 4 analysis side by side like that!
So the stronger AI is more decisive in the early game and doesn’t evaluate move 50 wrong, but besides that they are quite similar.
