It seems he agrees that there are some things still quite confusing about the new AI stuff, also some bug with the timeout timer and a problem with the scoring algorithm (AI-review review at 20:20 and 51:37, the other bugs just before the reviews)
I think the top three moves is seriously failing to show the top three moves. It might seem to people that it’s ridiculous, as it sometimes
- Shows the biggest mistake is actually the best move on the board
- Shows the biggest mistake was a move with a positive win percentage
- Shows that the biggest move is at a position it barely has read out
- Shows three moves that were played very quickly after each other (move 78, 83 and 84 or something)
- Reads out ridiculously long sequences that are not likely to actually occur in a game
I believe that this will harm potential future subscriptions, as the real deal is of course the AI review option for supporters, and this will give them very low expectations for an otherwise good feature.
So I suggest making some changes:
First of all, the way it finds the three moves does not make sense to me. It makes a “hunch”, and then reads out the moves from that hunch, but this of course does not improve the accuracy of the hunch. Since its hunch fails quite miserably quite often, it doesn’t really matter that the three moves are then read out. A far better system would be if it put all the effort in the hunch and not read them out as much, instead of reading out a very bad guess.
Secondly I feel that the AI reading out 18 moves ahead but only show a single variation is not very helpful either. It has to be a very special case where there is a forced succession of 18 moves such that those moves are actually “the only move”. Far more likely is that only the first three or four moves give insight in the situation, and the moves after that are just rough speculation or wishful thinking by Leela. Naturally Leela doesn’t base her judgement only on this 18 moves, but on a lot more branches.
Thirdly, the experience can be confusing to the player. It is not very clear whose move is the wrong one. I need to make several steps in my thinking: first checking what the last move was, then concluding that it was the other player who is about to make the mistake, and finally search for the letter A (which isn’t always easily found, since sometimes it’s other letters that are bright green). I’m sure this could be done better, for example by putting the actual mistake on the board, put a red cross on top of it and making the main suggestion by Leela stand out more.
Fourthly, I believe it would also be good to impose some balance in how close together it allows mistakes to be. I can foresee that it will ping-pong through consecutive moves, just because both players ignored a very important move elsewhere on the board. Of course this is helpful to be pointed out once, but if it’s done three times, that kind of wastes the potential of the move suggestion. So perhaps it would be interesting to spread the moves out over the game, say finding one biggest mistake in the each 1/3 of the game?
Fifthly, sometimes there is only “one move” or only “two moves” that work. It shouldn’t show any of the barely explored sequences. I would suggest not to show any of the sequences that isn’t at least explored a little more than, say, 5~10% of its total number of playouts.
Sixthly, it would indeed be nice if the sequences are loaded in the gametree, so you can click through them. It also becomes possible to feature several braches instead of just the single one.