Score estimator bug for NZ rules

Maharani · July 16, 2022, 9:39pm

Can’t remember if this is a known bug or not. The score estimator for demo boards does not take captures into account because it uses territory rules. A finished drawn game in which black has captured 8 more stones than white will be estimated as W+8. I’m using New Zealand rules if that’s relevant.

AdamR · July 16, 2022, 10:29pm

So the issue is that that the score estimator DOES take captures into account while it shouldn’t? I am a bit confused by the wording, but NZ uses area scoring so captures should not matter, right?

Maharani · July 16, 2022, 10:44pm

Oh duh, fixed - it would be W+8 in this case. Score estimator uses territory scoring, but NZ rules are area scoring.

Example: KataGo 7 komi self-play - game 82

Feijoa · July 17, 2022, 5:37am

I don’t know if it’s known, but it’s definitely wrong, and not just for demo boards. Here’s another example, from a real game that went to scoring:

Black has 9 more captures, AI score estimate B+26.6, level IV AI review B+36.0, weak in-game estimator B+34 or 35, actual score B+35.

(You can access the weak score estimator by forking the game.)

Feijoa · July 17, 2022, 5:54am

(I’m pretty sure that’s relevant.)

Maharani · July 17, 2022, 7:50am

Edited both posts again for clarification.

Feijoa · July 17, 2022, 2:58pm

In case anyone wants to look into it, this line is probably relevant:

github.com

online-go/goban/blob/7f836603f879038671c023278f466af6adfd02bd/src/ScoreEstimator.ts#L413

      
        
                                for (let x = 0; x < this.width; ++x) {
                                    score_estimate += res.ownership[y][x] > 0 ? 1 : -1;
                                }
                            }
            
            
                if (!res.score) {
                                res.score = 0;
                            }
            
            
                res.score += 7.5 - komi; // we always ask katago to use 7.5 komi, so correct if necessary
                            res.score += captures_delta;
                            res.score -= this.engine.getHandicapPointAdjustmentForWhite();
            
            
                this.updateEstimate(score_estimate, res.ownership, res.score);
                            resolve();
                        })
                        .catch((err: any) => {
                            reject(err);
                        });
                });
            }

It’s adjusting the score estimate for Japanese rules, which is kind of strange since the score estimator already understands Japanese rules - and if it didn’t the Chinese rules would have been a fine estimate already, without adjustment. Likely related to that is some old unused code above that was counting points based on “ownership” instead of using the actual AI estimate. Maybe all that was necessary before we switched to KataGo?

Anyway I’m guessing that since this code is fudging the score for Japanese rules, the score estimator does some reverse fudging to account for that, and instead of writing if(rules == 'japanese') someone wrote something like if(rules != 'chinese').

Maharani · August 4, 2022, 3:49am

Black estimated +11.3 for this draw. KataGo 7 komi self-play - game 84

Maharani · May 13, 2025, 10:41pm

Bumpity @anoek ?

square_fuseki · May 14, 2025, 4:12am

_

square_fuseki · May 14, 2025, 4:26am

uploaded to sgf library

but

forked game with
now, its correct

~~estimator probably just can’t understand that unusual sgf with a lot of comments and branches…~~
or it has something to do with New Zealand rules

Maharani · May 14, 2025, 2:00pm

It’s not that, because it’s universal across all of my games.

See also this real-game example from earlier in the thread:

square_fuseki · May 14, 2025, 2:44pm

I opened it with text editor and changed NZ to

RU[Chinese]

and uploaded again

now, it works

Maharani · May 14, 2025, 6:58pm

Seems pretty conclusive. Maybe a mod could change the topic title to “Score estimator bug for NZ rules”?

Feijoa · May 16, 2025, 12:36pm

Done!