Score estimator bug for demo boards

Maharani · July 16, 2022, 9:39pm

Can’t remember if this is a known bug or not. The score estimator for demo boards does not take captures into account because it uses territory rules. A finished drawn game in which black has captured 8 more stones than white will be estimated as W+8. I’m using New Zealand rules if that’s relevant.

AdamR · July 16, 2022, 10:29pm

So the issue is that that the score estimator DOES take captures into account while it shouldn’t? I am a bit confused by the wording, but NZ uses area scoring so captures should not matter, right?

Maharani · July 16, 2022, 10:44pm

Oh duh, fixed - it would be W+8 in this case. Score estimator uses territory scoring, but NZ rules are area scoring.

Example: KataGo 7 komi self-play - game 82

Feijoa · July 17, 2022, 5:37am

I don’t know if it’s known, but it’s definitely wrong, and not just for demo boards. Here’s another example, from a real game that went to scoring:

Black has 9 more captures, AI score estimate B+26.6, level IV AI review B+36.0, weak in-game estimator B+34 or 35, actual score B+35.

(You can access the weak score estimator by forking the game.)

Feijoa · July 17, 2022, 5:54am

(I’m pretty sure that’s relevant.)

Maharani · July 17, 2022, 7:50am

Edited both posts again for clarification.

Feijoa · July 17, 2022, 2:58pm

In case anyone wants to look into it, this line is probably relevant:

github.com

online-go/goban/blob/7f836603f879038671c023278f466af6adfd02bd/src/ScoreEstimator.ts#L413

      
        
                                for (let x = 0; x < this.width; ++x) {
                                    score_estimate += res.ownership[y][x] > 0 ? 1 : -1;
                                }
                            }
            
            
                if (!res.score) {
                                res.score = 0;
                            }
            
            
                res.score += 7.5 - komi; // we always ask katago to use 7.5 komi, so correct if necessary
                            res.score += captures_delta;
                            res.score -= this.engine.getHandicapPointAdjustmentForWhite();
            
            
                this.updateEstimate(score_estimate, res.ownership, res.score);
                            resolve();
                        })
                        .catch((err: any) => {
                            reject(err);
                        });
                });
            }

It’s adjusting the score estimate for Japanese rules, which is kind of strange since the score estimator already understands Japanese rules - and if it didn’t the Chinese rules would have been a fine estimate already, without adjustment. Likely related to that is some old unused code above that was counting points based on “ownership” instead of using the actual AI estimate. Maybe all that was necessary before we switched to KataGo?

Anyway I’m guessing that since this code is fudging the score for Japanese rules, the score estimator does some reverse fudging to account for that, and instead of writing if(rules == 'japanese') someone wrote something like if(rules != 'chinese').

Maharani · August 4, 2022, 3:49am

Black estimated +11.3 for this draw. KataGo 7 komi self-play - game 84