After a recent game involving a ko fight, I noticed that the post-game score estimator seemed way off. I was able to replicate it on a demo board:
Black just played (and captured) at D5. White can’t possibly have any ko threats here, right? But the AI gives the ko to White anyway. What’s going on?
Strange. The version of Katago on my computer, which is supposed to be less powerful than the one used on OGS, gives the correct estimate (that Black wins by 1.5).
Edit:
I guess the demo board uses Japanese rules with 6.5 komi.
W has 24 points territory + 6.5 komi = 30.5
while B has 31 territory +1 capture = 32
So that makes sense.
But white can’t even win by 2.5? Even it black answers a “threat” and white wins the ko, that’s just extra point, so that it’d be B+0.5?
There is something a bit odd indeed.
Similar idea, 6.5 komi, and the estimate is off by a point when a ko is on the board. Also the estimator is given the dame to white for some reason in Japanese rules.
and then filling the ko and dame
The demo uses Chinese rules with komi 7.5. If White wins the ko then D5 and E5 become white instead of black, so the difference is 4 points. Since 1.5-4 = -2.5, this is coherent.
So it does 
Now that I think about it, the fact that it’s unsure is even more surprising than letting White simply win the ko. It must be thinking that there’s something else that can happen on the board. But even removing all possible threats doesn’t help:
Note that when setting up these demos you should make it so the win depends on who wins the ko. Otherwise it’s harder to claim that giving it up is the “wrong” move.
Edit: also, the AI score estimator understands Japanese scoring now? When did that change?
I wonder if the score estimator is implemented such that it only queries the backend with the given position without providing the history. If you simply assume the configuration of stones is given as a starting position without having any history, then capturing the ko is legal.
That’s what I thought at first, but in that case white should win with 100% certainty, which seems to not be what happens when it’s set to Japanese rules.
Maybe the estimator tries to do something different for Japanese rules. I dunno. 
These cases are weird, because they should be trivially solvable, so there’s a good chance there is something odd about what queries are being performed, or how the output is being interpreted.
