Testing autoscore algorithms

Here’s a theoretical question I’m wondering about. In Vsotvep’s example above there was a dead stone marked in a strange place:

image

Is this kind of situation - specifically, a dead stone existing in dame - ever a valid thing to find in a properly finished game? Or is it always a sign that there’s something unresolved?

I think the players should explicitly express if they consider that marked white stone dead or not. Whatever they agree upon holds, and the game should be scored accordingly.

If they disagree on the status of that white stone, they should resume the game. If resumption is not working out to settle the disagreement, they should call an arbiter. The arbiter may choose to have an AI settle the issue, or they can opt to settle the issue by themselves.

1 Like

I understand, but are there actual cases in which experienced players would agree to leave a dead stone sitting in dame?

I’m not sure what you mean exactly. Taking it as an OTB game, do you mean they leave a dead stone on the board while rearranging the board to count? I think that would be quite strange behaviour for experienced players. I’d assume they made some mistake and they will fix it when they realise their mistake.

As I was recently reminded here, under Japanese rules, dead stones are only removed from territory. So a dead stone in dame - due to some kind of seki position - would not be removed. But to me it seems like that’s just an indication that they didn’t quite finish the game.

I’m wondering if there’s some kind of weird situation in which there might be a dead stone that should remain in the dame region of a seki, or if the situation indicated above is always an indicator of an unfinished game.

1 Like

I personally like how some servers dont help players at all with stone removal, and make people manually mark the dead stones every time. I think the main problem is that people dont always actually finish the games properly, either by pure accident or because they dont know any better. And theres some edge-cases where stones can be dead while sharing a liberty with a living stone, which may not be yet fully connected

White should of course play R1 before passing, but its not unusual to see stuff like being left on the board by both humans and ai’s

3 Likes

Ah, I understand now. Indeed that black stone at J7 should stay on the board during scoring, so white should capture it before passing. Not doing so is a mistake by white, but I would still consider the game finished if white opts to not resume the game after realising their mistake (and by doing so, agreeing that the J7 stone remains on the board during scoring, not counting it as a prisoner).

1 Like

Indeed. I think this (leaving unsettled stones to scoring, without requiring players to mark dead stones) is the actual problem that defeats any autoscore algorithm. And it mostly occurs in games of weaker players. For (stronger) players who finish their games properly, autoscoring wouldn’t be much of a problem.

Edit: I have to admit that recently I almost left unsettled stones to scoring myself, in Tournament Game: Fast Round Robin Handicap Tournament (100593) R:1 (gennan vs yebellz). I only realised my mistake after move 235 and I fixed it at the last moment on move 237 after my opponent had already passed.

Taking the example in the first post with bottom right black territory. That’s a simple case of a single contiguous region, with no dead stones, bordered by only black stones. So it’s simply black territory from the graph theory approach (or a human glance). But the algorithm first asks KataGo which discovers the cutting point weakness (which neither player saw so is irrelevant to scoring) which brings the 5 stones status into unsettled (which is wrong), and then the follow-up part of the algorithm fixes that wrongness, correctly, to give them back to black and score them as alive and the area as black territory. But by getting it wrong first, and then having to fix it, you make a less robust algorithm that’s going to trip up on other more complex cases (like the J1 case). You want the algorithm to solve the simple cases simply.

1 Like

What exactly do you consider a scoring issue? What are you expecting / hoping for from an autoscore algorithm?

As far as I’m concerned, the main argument against autoscore (except for difficulty of implementing it well), is that I believe that players should learn how to finish and score a game on their own anyway, because its a fundamental step towards understanding the game better. Although it increases the initial learning hurdle of go, i personally would not want to fully automate the scoring phase.
Instead, i would love if players mark the status of groups manually, and once the score is accepted, in case there are unsettled areas, the AI automatically points this out to the players (without changing the result).

2 Likes

AI shouldn’t point anywhere in my own opinion. Well not before that the scoring is closed and accepted by the players.

1 Like

Do you mean to say that this is an ambiguous situation in regard to the graph theory approach?

I don’t think it is ambiguous, as long as the players (or autoscore) explicitly determine the status of black’s group before the graph theory approach is applied.

  • If both players correctly mark black’s group dead after passing, it should be removed and the resulting region should be scored as white territory.

  • If neither player marked black’s group dead after passing, black’s group should be scored as alive with 2 points of territory. It’s what the players agreed to.

  • If the players disagree about he status of black’s group after passing, they should resume when it’s Chinese rules and call an arbiter when it’s Japanese rules.

4 Likes

I agree. Novice players need to learn that claiming dead stones is an integral part of scoring a game. This means that they need to have some notion about group status before they can properly finish and score their games.

Autoscoring could assist novices by declaring group status for them (without possibility for appeal or resumption), but that will still fail when stones are unsettled, as is bound to happen a lot in novice games. As a workaround, autoscoring could perhaps always declare unsettled stones to be alive. That would avoid novices being unable to finish their games, but I’m afraid it would also contribute to many novices never learning to distinguish between different group statuses, and also mis-scorings where AI notices a group is unsettled by some deep variation, while both players tacitly assumed it to be dead (perhaps even quite reasonably for their level).

Do you mean in the post-mortem AI review?

1 Like

After the game is finished, including the scoring phase.

  • moderator attention required to handle scoring mistakes
  • confused Reddit posts about bizarrely scored games
  • does not depend on whose turn it was before passing
  • does not reveal surprise invasions (as much as possible)
  • just decides life/death and then follows the (simple graph theory) rules of Go scoring
1 Like

Sure, that works for the simplest cases, but as soon as there are unsettled stones of each color around you end up with a giant surprise seki, not so different from a surprise invasion.

1 Like

Yes, I think leaving unsettled stones to scoring and then letting AI decide group status and the resulting score will inevitably lead to some problems.

I’m sort of playing the Devil’s advocate here, trying to come up with potential benefits of autoscoring for weaker players, while actually preferring that they learn to claim and agree on dead stones without depending on AI assistence.

2 Likes

Well, I did acknowledge this preference at the beginning but still want to explore possible autoscore algorithms. So,

I made a new v2 version that decides the status of unsettled stones based on counting live stones along the boundary of the region. (Stones bordering only the unsettled part of the region do not count for this purpose.) Here are some examples from above that I think it handles better; click images to see the details and try it for yourself:


image


image

Also I added a new feature to “unsettle” any dead stones in dame before deciding life/death. That helps with the cases where the dead stone itself is creating a surprise seki like the one we discussed earlier. Here’s another example from above where a dead stone is promoted to being alive:


image

(Note that the whole left side here is still dame due to an unclosed border near the top.)

Please let me know if you can find any weird cases where v2 gives a bad result.

To me, the policy to decide unsettled group status by the majority number of stones touching the adjacent unsettled region seems just as arbitrary as declaring all unsettled stones alive. I would have no issue with declaring black’s stones alive (in seki) just from the fact that white failed to kill them, regardless of how many white stones happen to be adjacent to the bottom region.

Are those black stones unsettled? I don’t see how black can live there even if black starts.
Edit: I’m sorry, I misread. They are unsettled. In that case, it looks like both players assumed black is dead. But I think it’s pretty hard to come up with a simple algorithm to detect that. Anything less than inferring players’ thoughts like that would be rather arbitrary IMO. This case in particular should be left to the players’ judgement about which stones they consider dead.

In my view, that black stone was originally unsettled (not dead). I think promoting unsettled stones to alive is usually better than demoting them to dead. If that leads to a large region becoming neutral, so be it. It would be similar to a not-closed region like the left side.

3 Likes