Not an error, just an interesting oddity in the data: this game appears to have a komi of “64.64”. Which is neither an integer or half-integer. Kilpperts Oxy (note: not to be confused with CHOWpperts Oxygen)
3 Likes
Interesting game chat, too…
1 Like
Has anyone attempted to generate tsumego problems from this SGF dump?
I’m specifically interested in creating problems of ladders and nets. So far I’ve looked at GoTools collection of computer generated ladder problems.
I also saw that @hexahedron implemented a ladder search for KataGo, I wonder if that could be used to search the database of games to find positions automatically. Maybe even combine with KataGo itself to evaluate how “interesting” a position is, to exclude very basic positions.
1 Like
I suggest you to open a new thread for this to get more attention.
@za3k, are the updates, which you talk about in the first post of this thread (Can we get an SGF database dump?), being made with which frequency? And are you able to edit that first post, to change it with the latest ones? The Internet Archive thing still has 2021, and the files there start in 2013 - so, as pointed in this thread, there are also many games before that are not included.
I tried to access your home server a few times, now, and I just receive “504 Gateway Time-out” errors. The archive.org is there, normally.
When you update that collection, write in the update notice the date when the whole game set was read. For now, I want to have no 504 error to a possibly more complete dataset. I see posts here from the last 60 days, so there should be not many spiderwebs around…
Best regards. 
I have come across another issue.
In some game records, like game 2602573 or game 3647667, the player names include backslashes as their last character. These backslashes are not escaped in the SGF conversion and mess up the file:
…PB[diegoluc\]PW[Kimsbike]BR[8k]WR[8k]…
SGF-parsing programs like Sabaki and KataGo see this and think that black’s name is diegoluc]PW[Kimsbike and that there is no white name specified.
This is indeed the proper interpretation, as some players legitimately use ] in their name. For example, this is represented in SGF as PW[[V\]oR0BuSHek].
If you should update your dataset in the near future, it would be great if you could have backslashes in names consistently escaped, for example like this: PB[diegoluc\\]. 
3 Likes
Looking at the SGF games, I have a bunch in there that look weirdly corrupted (I’ve been looking at sgfs-by-date.tar.gz. I get a bunch of invalid moves (single suicide stones, playing where there is already a stone, obviously bogus handicap placements) and when I compare to what I see on the live site, many moves are sort of near where they should be, but a few intersections off.
Have other people worked with the SGFs, or just with the JSON files? Because it seems quite wrong 
For example (there are thousands of these):
sgfs-by-date/2008/11/14/122198-Sethius-GnuGo.sgf:
(;FF[4]CA[UTF-8]GM[1]US[lightvector/ogstosgf.py]PC[OGS: https://online-go.com/game/122198]DT[2008-11-14]PB[Sethius]PW[GnuGo]BR[30k]WR[20k]GN[Challenge from Sethius]TM[259200]OT[86400 fischer]RE[B+2.5]SZ[9]KM[6.5]RU[Japanese]HA[3]GC[correspondence,unranked]AB[dd][ed][ge];W[ee];B[cg];W[de];B[be];W[ce];B[bd];W[cd];B[hg];W[gf];B[gh];W[dg];B[hf];W[fg];B[ch];W[cf];B[bf];W[dh];B[hd];W[hc];B[fh];W[eh];B[ih];W[fi];B[hi];W[ic];B[ah];W[bc];B[ac];W[ab];B[ad];W[bb];B[ff];W[eg];B[fe];W[fd];B[ef];W[df];B[id];W[gd];B[he];W[ci];B[bi];W[];B[di];W[ei];B[gi];W[];B[ci];W[];B[])
Loading from SGF, the handicap moves are:
= ABCDEFGHJ
9·········9
8·········8
7·········7
6···○○····6
5······○··5
4·········4
3·········3
2·········2
1·········1
ABCDEFGHJ
Resulting in final game:
= ABCDEFGHJ
9·········9
8●●·······8
7○●·····●●7
6○○●○○●●○○6
5·○●●●○○○·5
4·○●●○○●○·4
3··○●●●·○·3
2○·○●●○○·○2
1·○○○●●○○·1
ABCDEFGHJ
Game on the site right now, from SGF download: Challenge from Sethius
= ABCDEFGHJ
9·········9
8○○·······8
7●○○···○○○7
6●●○●○○○●●6
5·●○○○●●●·5
4·●○○●●○●·4
3··●○○○○●·3
2●·●○○●●·●2
1·●●●○○●●·1
ABCDEFGHJ
So in this case white and black are switched around, and handicap stones are wrong. But I’ve also seen games where normal moves are also off for one or both of the players, but not every move.
When I use the ogstojson.py on the JSON from ogs_games_2013_to_2021-08.json.gz I get an SGF that matches the current game. So I reckon something must have gone wrong when SGFs were being generated and some non-trivial fraction of them ended up subtly broken.
Luckily, at least based on this game it looks like the original JSON (in the dumped archive) is correct so I can attempt to regenerate the games:
= ABCDEFGHJ
9·········9
8○○·······8
7●○○···○○○7
6●●○●○○○●●6
5·●○○○●●●·5
4·●○○●●○●·4
3··●○○○○●·3
2●·●○○●●·●2
1·●●●○○●●·1
ABCDEFGHJ
Anyway, beware!
2 Likes
Maybe you ran into the same issue I posted and fixed for handicap games further up in the thread in November? As far as I’m aware, za3k hasn’t regenerated the sgfs since then.
I think the converter also fails to consider the occasional move color override that appears in some records:
We only look at coordinates it seems:
It’s relatively rare of course, maybe just in games that are started from uploaded SGFs (I’ve seen it in some games with an interesting artistic selection of starting position) or due to site bugs (I’ve seen it in some old Fuego bot games, maybe coinciding with errors key being present). And a lot of instances of this are the corrupted games (one of which you pointed out above) where it looks like it was a 9x9 game or something, but there is this one single move outside that area and is now displayed on 19x19 board (eg: Vriendschappelijke Wedstrijd - possibly also SGF upload?) But throughout the dump it’s still thousands of files.
I’m also not sure if the goban implementation on the web site itself handles the optional color specification correctly. It seems to do so in some games:
2 Likes
This collection has been updated with games from 2021 to 2025-05. The number of games has approximately doubled, from 27 to 56 million games.
It’s available in the same formats as before, on my website (link on first post):
- JSON (one per line) of each game, unmodified from OGS
- SGF files (one per game), in .tar.gz files. There are two downloads – one organized by date, one by player. Bots are excluded from the player collection
- Uploaded games (0.7% of games) are excluded temporarily because I discovered a privacy concern, but they’ll be up eventually
I’ve made a best effort to fix any problems people mentioned to me the first time around. Weird names shouldn’t cause any bugs, and I think ranks in the SGFs should match the ones displayed on OGS now. (“[?]” rank will display as OGS’s best guess at the time)
Feel free to report (or fix) any problems over at GitHub - za3k/ogs: Crawl online-go.com and package up the results for the public , which includes the crawler and SGF converter code.
As before, I probably don’t have a lot of time to answer “how do I…” questions on what to do with the data once you have it. But I’ll keep an eye out if people spot any problems in the actual dataset (missing games, wrong ranks, that kind of stuff).
Everything is a bit more streamlined now, so hopefully I can keep things up to date more going forward.
Thanks to hexahedron, anoek, siimphh for help on this update!
P.S. If a mod could update the top post with the updated year range that would be great (they apply to the za3k.com link but not the internet archive, which is still stuck in 2021 for now). Also copy+paste the github link I guess?
P.P.S. @anoek you or other OGS devs might find this list of interest – all of these database entries appeared to me to have some problem with them ogs/game-blacklist.txt at master · za3k/ogs · GitHub . The most common is a winner who isn’t one of the players.
5 Likes
That kind of sounds like rengo.
I think the last one on the list is a rengo anyway
71822605 - 牢p无敌,服了
Edit and the one before it
70640302 - testoid.ru vs. DAMIRx
1 Like
Some of the entries are rengo, some aren’t.
@za3k I’m curious if you have any plans to either upload the 2025 update to the Internet Archive or get your personal server back online! I’d love to grab the newer version. Thanks!
My personal server has been back online for a while – you’re welcome to grab it now. The IA upload was delayed due to some privacy issues – the crawl grabbed some games that really shouldn’t have been available, and then I never really heard back about whether the issue was fixed.
When I do the next update, I’ll put it on IA too this time. Should be in 2026, probably – update is in progress.
2 Likes