Now that we have so many open source and strong public Go “AI”, I thought that I would use the latest 10 block network of Leela Zero to do some analysis of the famed “God move” played by Lee Sedol in match 4 against AlphaGo.
https://archive.org/download/LeelaZeroGod/move78_logs.txt
https://archive.org/download/LeelaZeroGod/logs_black.txt
https://archive.org/download/LeelaZeroGod/logs_white.txt
https://archive.org/download/LeelaZeroGod/game4_lz.sgf
This was move 78, where Lee played at the L11 as white, turned the tables and ultimately won the game. So I load the sgf to move 77 into Sabaki, just after AlphaGo played its move at F17 as black, and then apply the attached Leela Zero engine and let it analyze it as white for a good 5 million playouts or so.
Leela Zero never picks the “god” move, even after millions of playouts it never even considers it nor explores it as a possibility or variation. Hence it’s fair to say in a real game against Leela Zero, it would never have played at L11 as white. Either the move was no good, or it was very good but LZ was blind to it.
The next question is then how good really is this move? To explore that I ran two instances of Leela Zero to have it play as black and white. Not too unsurprisingly, its immediate follow up moves after move 78 as black mirrored that of the moves AlphaGo had played in the match, even after a million playouts the moves matched so I was fairly certain the LZ tracked AG for that segment. Then for white I played where Lee Sedol played in the game.
It wasn’t until move 83 that there was a divergence. In the actual match AG played at N8 as black on move 83, whereas giving Leela Zero two million playouts and plenty of thinking time it choose L14 for the same move. Interesting to note is that upon review of LZ’s logs, it appears that it favored L15 for move 83 all the way up to 1.7 million playouts where it then switched to L14 as the best move. Although LZ didn’t choose AG’s N8 move, at least it considered it and thought it had a good winrate, but didn’t go for it. (contrast this to Lee’s move where LZ didn’t even see it after a good 5 million playouts!)
Obviously since we can’t know how Lee would have played out in the game with this change, I substituted instance 1 of LZ as Lee and identical instance 2 of LZ as AG. My thinking is that if the “god move” is really that good, then by this point on this should be a won game for white, all else being equal…
My thinking is since by move 77 Leela Zero already predicted a 80+% winrate for AG (black) and since Leela Zero absolutely failed to see Lee’s “god move” on move 78, and since the subsequent AG moves tracked LZ’s moves until move 83, this was the perfect condition to test whether or not Lee’s move 83 truly was a great unseen move. Because of the fact that at very high playouts the Leela Zero already is a top level pro, and at the beginning prior to the god move it was obvious white was losing badly, then by pitting LZ as Lee Sedol vs LZ as AG and against one another at very high playouts (2 million per move) then if the outcome is in favor of white, this pretty much substantiates the quality of Lee’s move 78.
Note: it’s interesting that the LZ instance as black after move 78 didn’t have the sort of meltdown going crazy and playing silly moves occurrence that happened with AG in the real match. My guess is either the “god’ move really wasn’t that godly, or AG sandbagged and purposefully let Lee win one game out of mercy PR when it saw the chance, prob since Deepmind may have felt that a 5-0 may actually get them negative PR blowback. Maybe a combination of all of the above. But it’s interesting that the god move was not enough to win the game and it did seem that AG played far more poorly after move 78 than compared to even Leela Zero 10-block network played without thousands of GPU’s or TPUs.
Or in the alternative I could be wrong about the entire premise and maybe it is the case that LZ on very high playouts is already stronger than AG Lee version.