The integrated AI Review feature for OGS

#1

Hello OGS!

In the near future all 19x19 games will be analyzed and “reviewed” in part or in full by Leela Zero, ideally helping to identify great moves or big blunders for your edification.

You can see what this will look like and how it will work by heading over to the beta site. For an example of a game that has already been reviewed, see: https://beta.online-go.com/game/3451

For a “full” review, there will be a chart showing Leela Zero’s estimate about who is going to win, and clicking anywhere on the chart (or navigating to any move normally), you will notice several places on the board that Leela considered good potential moves. Clicking any of those will show what she thought might be a good sequence following that move.

The other type of review is a “fast” review, where the top 3 game changing moves are identified and analyzed.

Because of the resource intensive nature of these reviews, all 19x19 games will have a fast review, but only site supporters will have full reviews generated (although viewing the full review will be available to everyone). (Note that on the beta site, everyone can do full reviews for testing.)

The interface is still a little rough around the edges, but the core functionality is ready to check out and give feedback on.

54 Likes
pinned globally #2
#3

Awesome!

1 Like
#4

Isn’t this gonna be a ton of games? How is this even possible?

Impressive!

1 Like
#5

Sounds really cool!

I assume this review only becomes available after the game is over, and does not happen in real time while in progress?

By the way, after going to the “Top 4 moves…”, clicking on the “Full AI review” link causes an error popup that says “Permission denied”

1 Like
#6

This will be a truly wonderful addition!

1 Like
#7

Yep :slight_smile: Lots of worker nodes analyzing the games together.

Correct, only after the game is over.

Shoot thanks! I think that’s because you weren’t logged in, I’ll fix the error message.

4 Likes
#8

I honestly didn’t even know that there was a beta site. This is amazing! Do I need to make a new account for the beta site? It won’t let me login with my username/password. That said, congrats on the new groundbreaking feature! <3

2 Likes
#9

Yep you’ve gotta create a beta user, it’s effectively a complete and isolated system from the main site

2 Likes
#10

This sounds awesome anoek! I figured when I saw you post in the LZ github a while back you were working on something neat. :slight_smile: The way lichess has stockfish integrated I always thought was really cool.

4 Likes
#11

I knew it was coming as soon as @anoek posted this :slight_smile: :

3 Likes
#12

Hello! Very nice new tool! I think it’s the first Go server that has one like this, it well be amazing!

I’m wondering what’s the difference between the short review and the full review. Can you tell me?

4 Likes
#13

The short review only does the top 3 moves, the full review does the entire game

5 Likes
#14

Huge thanks!

2 Likes
#15

This is amazing!

My only (totally unsolicited) suggestion would be to consider using current 15-block weights for LeelaZero, such as network edb61bc2 (aka LZ-15b-202). This is because you’re not gonna read any ladders at 200 playouts, and 15x192 weights can go up to 550 playouts using the same resources, improving accuracy at slight cost to overall strength.

7 Likes
#16

Can sfg’s be uploaded upload for review, or does the game need to be played on the betasite as well?

Can’t wait to try it as see how it helps my game improve.

2 Likes
#17

That’s an interesting observation. I was planning on doing more playouts for the production site, though I’m not sure how many are necessary to read out ladders to be honest. Would 800 be enough, and would 800 on the 40 would be better than the ~2k of the 15? Any thoughts on what some good limits / values would be?

No support for uploaded games yet, I’ll be watching our resource usage and seeing if we can afford that.

2 Likes
#18

How can you tell the difference between a blunder and a good move

#19

Yes! I’d avoid using a 40b net at 800 playouts and instead suggest a 15b network at 1000 playouts or more. I set up a classic cross-board ladder below (joseki used: http://josekipedia.com/#path:peqcocqeqfpfoeqgrfrergqhpgofphrh):

40-block network 223 needs 1400 playouts to see that the ladder is good for White (>50% win rate):

40-block network 211 (OGS beta) actually fares better in this test. It needs 1200 playouts:

15-block network 202 (my link) does significantly better, kind of shockingly. It needs 900 playouts:

Based on this test, I feel that anything under 1000 playouts would be unreliable for reading ladders, and it sounds like playouts in excess of 1000 are only achievable using a network smaller than 40b. So I’d go with the linked 15b network (or its successor, when that comes).

3 Likes
#20

H6 was a -10% blunder. I found it by clicking on the graph at a place where the line dipped suddenly, then clicking the left and right arrows till I was precisely on that turn.

You can see on the board that Leela thinks white should have played P10

3 Likes