Yes, that is right.
My understanding is that the use case for AIAMS is that we want to get a fair and helpful scoring even for games that are unfinished. I mean otherwise we could just use auto scoring. That’s why I wanted to look at this example of an unfinished game. I personally think the way this is scored seems unexpected.
what about having algorithm that determines if game is 100% properly finished or not?
if yes, use one logic of scoring
if no, use completely different logic of scoring
Yeah that game is still little bit unfinished… Whoevers turn it is, they should play C1, its worth 1 point. Then white should play at A7, and after that white can play at A9 which forces black to answer on D9, this D9 itself is not a point for black. So, no seki there ^^
It seems ok to me. As you said we need to score games that are unfinished according to perfect play. Scoring according to what could happen, via Katago leads to weird results where connected areas are only partly counted as territory which is no good.
With that in mind, it seems ok to not mark A7 as a point, since it’s only one point. However, you can’t not mark D2 as a point since E2 is a point (even in the throw-in variation, there’s a point there). D2 could be a point, if white C1, so it’s kind of 50-50, may as well go for the logic of keeping connected areas scored.
While black could lose a point when forced to capture with D9, again there’s no real way to discount that without removing a point from a connected territory.
So I’m not sure this position can be scored any differently really. In over the board play it should be resumed, but online I guess this is fine if both players accept it, if not they can resume.
Second Observation
Shouldn’t the border at C1 be closed off? Why isn’t it marked red? Either W or B should play there to get a point.
The border is already closed off. There is still some yose that could be played, but that’s not a matter of closing the border (and not the purpose of the red marking).
Thankfully, as marking this kind of moves would be problematic.
You’re right.
Ok, now my last review spam in this thread. But two more things I saw while testing:
gennan has already mentioned that some border markings are unexpected. I personally found this one a little unintuitive:
And finally, not sure why the B stones in the bottom left hand corner are marked dead. I’ve seen groups that were obviously dead marked alive (can provide an example if wanted) just because it would have taken a few moves. But these groups are somehow dead?
I think this is wrong. Both players have already passed so they’ve agreed there are no more moves to make and ask the groups and stones are either obviously alive or obviously dead. There is no more “perfect play” left as the game is finished.
All objectively finished areas and groups can be counted but what to do with unsettled groups? The only reasonable option is to consider them live for scoring, that’s very likely what the players assumed and if they didn’t then they can resume to show otherwise. Any other indication would give a very strong and useful hint to one or both of the players.
Perfect play and katago imo are only relevant for determining which groups are live, unsettled and dead.
I guess it’s a matter of opinion but this seems fine to me!
And finally, not sure why the B stones in the bottom left hand corner are marked dead.
I agree with you here though. Not sure why it’s marked as dead?
I may have misunderstood but I think Shinuito just meant some games are “unfinished-according-to-perfect-play” (i.e. as a way to specify what he meant by “unfinished”). Not to suggest we should assume perfect play continuation to score them.
I even got confused now when rereading, so maybe it was just worded badly 
We do need to score unfinished positions, where unfinished has to be with respect to something, e.g. perfect play. However scoring with respect to that perfect play, like with Katago is problematic. It could lead to strange markings as to what is alive or dead or how much of a ”territory” belongs to one player or the other.
What about never scoring unfinished game?
If both opponents passed too early, then they would be encouraged to return or click “Let AIs finish the game”.
if both players click it instead of return, ghost black and white stones appear and game is always properly scored. No need to invent new scoring rules that are able to score unfinished game in not weird way.
It’s also possible that both players have assumed that an unsettled group was dead, or that one of them thought it was dead, while the other thought it was alive. Under area scoring rules, resuming is the correct procedure to settle life and death disputes. However, under Japanese rules, life and death is not exactly settled by further play. For example, in some cases, a group might actually not be killable under normal resumed play (such as due to unremovable ko threats), but it is declared dead by the Japanese rules, since life and death is determined during scoring using hypothetical analysis while applying different ko rules.
Technically, under Japanese rules, both players passing while leaving an unsettled group could put the game in an situation where it should be declared “both players lose”.
While, we’re talking about dealing with resumptions, we also have a bug with how game resumptions are handled under Japanese rules. Technically, whoever presses the resume game button should give their opponent the right to move first. This creates the awkward situations about how to properly deal with unsettled groups, since neither each player would want their opponent to request the resumption instead.
I can’t emphasize enough how much I do not want that. 
Right, there is a case where a group looks very dead but there is something slightly tricky to play to make it alive, so showing it as unsettled could also be a hint? Like there is some fun way to respond to a throw-in or maybe the under the stones thing that should be played out?
That’s a fair point.
In actual games it’s going to be so much more common that a group can be killed and neither player noticed, so that’s the more important hint to avoid.
So fundamentally the problem is that katago is much better than the players so whatever it says about the final game state could leak out superior reading information to the players.
The solution to that sounds like making sure the autoscorer reading is made weak on purpose so there wouldn’t be much to leak out. It should be a weak network with a single payout and we only make use of the determination if the confidence is high.
So we end up with a situation where groups are more often than not marked as alive when there is any doubt or any notable subtlety to the conclusion.
I don’t think there are many (any?) players that actually really want realistic Japanese rules on OGS. It really would be great if we could replace the supposed “Japanese” rules on the site with “Japanese-like” meaning territory scoring and Japanese handicap/komi… and anything else people really associate with Japanese rules other than those?
What if we had full manual scoring, but neither player can see what the other marks until they submit?
If B and W both mark the same, the game ends with the agreed score. Can’t be cheated since both mark independently.
If they disagree, the disputed group(s) could be highlighted, and play resumed.
That way we avoid all the problems of a clever AI trying to understand strange humans.
About the autoscore marking of black’s group in the lower left.
If the black player had considered it alive, I’d expect that they would have connected at A7 before stopping the game.
If the white player had considered it alive, I’d expect that they would have captured at A7 before stopping the game.
But neither player did that, which suggests that both players considered it dead. So I think the autoscore did a good job trying to guess the (wrong) group status adjudications of both players.
Looks like a great idea, what to do with seki in this case? Especially in the case of beginners.
What about seki? If one player marks a seki group dead and the other doesn’t, the game would autoresume, just like other disagreement cases.
Or do you foresee a problem in general when beginners keep disagreeing about some group status and get stuck in an autoresumption cycle when neither of them can see where the disagreement is (though in @BHydden’s proposal they would be informed about the location of the disagreement)?
Exactly this, yes, thank you. I think it means that we need to have three toggleable states of groups on the board – alive, dead, and seki, possibly, with some explanation to beginners what the third means, instead of just “marking groups alive or dead”. For better UX, we could also have a more distinguished marking of groups in seki, not just lack of squares, but maybe squares of a different color, to clearly highlight the third state of the groups.
I guess that would also be a hint in itself? If the autoscorer does not trigger then that means there is something unsettled you have not noticed.
So do I, but anoek decisively prefers to present open border like that, and also there doesn’t seem to be a clear consensus on how else to present open borders:
Opinions seem to be split about evenly between this (“fully covered”)…
…and something like this (“slightly fuzzy”)…
… or perhaps something like this (“realistic”)…
