A compendium of OGS's terrible scoring system confusing beginners

[Vsotvep]

@Vsotvep

Since the AI algorithm launched, the moderator team has been keeping track of incorrectly scored games in an internal thread, and I believe to have found an improvement to the current algorithm that solves 95% of the current mistakes it makes

For those interested, here’s some snippets from that thread with my improvement in demonstration.

Here's an example of a group that looks alive, but can be killed by continuing play

@Vsotvep

• F14 is an empty intersection and borders both settled black stones and settled white stones, and thus is dame.
• P10 / Q10 are similarly dame.
• The 7 intersections around F9 are not purely empty intersections, but contain stones as well. These stones are for the majority Black (5 Black stones vs. 0 White stones), thus should it be counted as belonging to Black to be scored “correctly”.

Alternatively, inside the 7 intersections the empty intersections at F8 / G9 could be evaluated using the dame test as well: F8 & G9 are not dame, since they cannot reach settled white stones without crossing over a Black stone.

Here's an example of open borders

@Vsotvep

The players are trying to score before the territory is closed off. (screenshot, since I’ve advised the players to resume the game, so it may not be visible later)

In this case, my proposed algorithm would do the following. The difference between Black’s turn and White’s turn is only M13 and M14. Since these touch settled Black stones (L14) and settled white stones (L13), the auto-score would mark them as dame. As a result, unless one of the players notices the problem, the score would be the average between what would happen if White went first and if Black went first.

Note that under Japanese rules (if the winner depended on it), neither player would want to resume the game, since the opponent may move first and hence “win” the territory. Officially such position would declare both players to have lost the game.

Here's one with a huge unsettled group

@Vsotvep

If White plays J4, Black is dead, due to shortage of liberties. Of course, if Black plays J4, Black is alive.

All of this is unsettled, but the Black stones have the majority, so should be considered Black territory. Not doing so may hint the White player that there is this killing move.

Here's an example of how the algorithm could fail, but it can be improved a bit by allowing a margin of error

@Vsotvep

In this one, it doesn’t work as well. Somehow, H9 and G8 are always marked as White, even though there is the H8 cut which should either kill H9 or the G8 stone. H8 is therefore the only “unsettled” point.

J1 and K1 switch colour, but J2 is always considered Black, which makes it a Black settled point, which is of course incorrect…

As a result, H8 is considered unsettled area without a majority of stones of either colour, thus marked as dame, and J1 and K1 are marked as White, since these do have a majority of White stones.

Proposal to fix this

Consider any intersection of which KataGo is very uncertain to be unsettled anyways. The Black / White squares that the estimator gives at these disputed intersections are so small that they could easily switch to Black or White just by random process in KataGo’s way of thinking.

E.g., the smaller squares in the following picture would qualify for being unsettled:

This would increase the number of unsettled stones, but that’s good. As a result:

Here both the J1 unsettled group and the G8 unsettled group would qualify as White territory, since it they both have more White stones than Black stones inside the unsettled area.

Finally let's also include a failure, although I would like to defend the AI here, since this seems impossible to score automatically without resuming the game

@Vsotvep

This would give the entire marked area to Black, for being the unsettled part with a majority of Black stones, which is clearly the wrong evaluation in this case.