The integrated AI Review feature for OGS

It does that sometimes when you load games via new tab/window. Refresh.

But yes, bugs are tracked via github.

2 Likes

My question was how to interpret a large positive value like 41.5%. In my view the full interpretation table is something like

-30% : blunder
-5% : mistake
0% : good move
5% : LZ admits it slightly misjudged the previous position
30% : LZ admits it severely misjudged the previous position

Do you agree with this table? If not, then how would you express the meaning of a large positive value like 41.5% in English?

Since I’m worried about a bug but you’re not, I investigated a bit more. When I analyze the example game on my computer with the same network and the same number of playouts (200), the winrate doesn’t change at all at move 42 (D13 0%). The largest positive change over the entire analysis is 8.8%, and this number drops to 6.7% with 2000 playouts.

This support my claim that seeing 41.5% in the preview and so many moves above 10% indicates that something is not right.

1 Like

I think you’ve got to curve the table, because win rate is nonlinear. For example, if my win rate is 29%, your table says I can never blunder, even if I lose 60-points in one move. Conversely, if the game is even, where a 2-point mistake is worth 10%, your table might flag a 6% loss as a mistake even if it doesn’t actually lose any points. I feel a table like yours should try as best as possible to emulate score, where a 4-point loss is marked as questionable, and you judge mistakes and blunders from there.

1 Like

I like very much this new feature and think it should be a wonderful tool to boost site supporters number. :wink:

Is it available an example of the “fast” review?

You can see an example of the fast review by clicking the little three line icon thing right above the win/loss graph, it says “Top 4 moves according to Leela Zero”

2 Likes

The fast review is very teasing, but having just a number isn’t much satisfying.

Could it be possible to have also the win-rate graph in a non-interactive version?
It would be enlightening to understand the flow of the game and also the importance of those Top 4 moves.

Another question:
is it normal to have a strongest move (bightest green) without variations?
See picture:

Nice feature :partying_face:. In future, I will get a slap in my face after each game :wink: Let’s hope it will help me improve.


It seems to me that the bar showing the chance to win for both players and the win rate change of that move are using the blue line, not the better purple line.
For example at move 42->43 the purple line is already near 100%, while the blue line is at about 50%. The improvement of blacks move seems to be only an artefact of LZ not reading at all.
Do I get something wrong or is that because I don’t have a supporter account on the beta site or is this a bug?

image

An improvement of 40% against LZ previous estimate is much much more than I would expect to be possible. At least after enough playouts.


As I understood it’s not the strongest move, but the move LZ explored most. It probably was the best move until LZ explored it in some length. After extensive exporation LZ probably decided it isn’t as good as she frist thought. With more playouts the color would probably be less intense.

explored = spend reading time
letters = strongest moves

2 Likes

Amazing :slight_smile:

Are you planning on crowdsourcing any of the analysis bots, like the way lichess does with fishnet? I’d be happy to contribute GPU time if it was as simple as running a docker container and plugging in an API key.

3 Likes

This is extremely good!

This whole “explored vs strongest” thing is puzzling me, possibly because seeing ‘A’ makes me think “I should play here”.

AIUI, the place that Leela would play at any given time is the one with the most playouts - the most explored. That’s the “blue” circle in Lizzie.

Am I understanding correctly that is the green coloured node in this feature?

It’s not totally intuitive that some other node than the one the bot would play is marked “A”.

I’m I misunderstanding?

GaJ

1 Like

I thought about that but I’m a bit worried of malicious users, we’ve had a few very technically competent trolls over the years… so basically to avoid that headache I figured it’d be best just to keep it all in house

3 Likes

I don’t understand the problem you are seeing.

After black 41 was played, white’s odds were 52.4 and black’s 47.6. Best next move for white is J13. If this move had been played, I guess odds would have remained the same.

Problem is white 42 was not J13 but A16, which is a mistake and changes the board. Hence best next move for became black D13, which greatly improved its odds.

Now it is true that given that black’s best answer to white 42 improves its odds by 41 percentage points, I would expect white 42 at A16 to be rated much worse than just -1 percentage point.

But in any case, Leela would not have played A16 and therefore did not misjudge the previous position.

The odds should be given as “if black and white will play the moves which improve their probability to win as much as possible starting with the given board, what is the probability for black/white to win the game”. With this assumption, neither black nor white should be able to increase their odds, but only decrease their odds by playing suboptimal moves.

So the move black 42->43 which increased blacks odds by over 40 percent points shows a big misjudgement of position 42 by LZ.

I cannot find the feature on iPhone, is this just for the desktop version or am I missing something?

The feature is not available yet. There is a preview available if you follow the link in the initial post.

2 Likes

I used the link but can’t see any reference to the AI review, no graph at all.

Is it possible and is it still a probability? I don’t know how Leela works, but following what you are saying, what should white’s probability to win be after Black 1? If all future best moves are factored in, the probability is not the probability to win the game but the probability to play all the best moves, is it not? My point is that the probability should probably factor in bad moves as well.

After black 41, whiet’s odds to win are 52.4%, with the given board. Whether its odds will increase or not depends on the quality of the next move. If the next move improves the board for white, then the probability to win increases.

So I still don’t really understand the problem that is raised.

The graph isn’t there any more. I don’t know why. Maybe @anoek changed something.

I know get sometimes the error message:
image
Maybe it’s related.
(My scriptblocker blocks fonts.googleapis, googletagmanager and ravenjs, but this wasn’t a problem before)

I idealized my explanation a little. I assumed LZ knows what the best move is. If LZ indeed would know what the best move is for all given boards, then LZ would give the odds only as black wins, draw and white wins.

LZ gives us only the odds, because it is not able to find the best move, but uses positional judgement and reading to find the probably best move. LZ cannot read through the whole game, so LZ factors in her own uncertainty of the positional judgement, but doesn’t factor in possible mistakes (possible mistakes in the sense of “I know there is a better move, but maybe I will not play it”).

The probability LZ gives are LZs estimations for a game LZ vs. LZ where both LZs play as good as possible.

The given probabilities are used by LZ herself to find the best move to play. To factor in blundering (hope your opponent makes an error) doesn’t do any good if she wants to win the game. I think the neural network factors in if the move leads to a hard to read positions, since the zero AIs tend to play very save if they think they are ahead.

The probability is not a statistical factor for random players, but only for a LZ playing at training strenth (1600 playouts).


In our case we ask LZ after white 42 “what is you probability to win the game as black if you would have to play against yourself?” and LZ answers: “The best move for black I can find has a probability of about 50% to win the game.”
Then we place the black stone 43 on the board and ask again: “what is you probability to win the game as black if you would have to play against yourself?” and get the answer: “with this move my probability to win as black is now 90%. I hadn’t seen that move before you played it. I have to admit, you found a move that is much better than I thought is possible.”

It statistics nerd in me forces me to write this here:

It’s a 41.5pp (percent point) change not 41.5% (percent).

But do like you want. “Percent points” will probably irritate many users because they aren’t used to it.

image

9 Likes