First of all, thanks @za3k for hosting the OGS game collection. It’s an awesome resource to have for go programming projects. I started looking at it the 2025 JSON collection recently, and I noticed that there’s only about 100 games from September 26, 2023 to November 20, 2023. Did something go wrong with the database crawl for these dates?
1 Like
We did overload the OGS API at some point during the dumping process and it was quite possibly at the point where we around 2023 somewhere, the dumper was restarted a number of times and produced more partial dump files that needed to be merged, and maybe something fell through the cracks somewhere in the process.
Just to make sure we’re looking at the data the same way, by “games from September 26, 2023”, those would be games started or ended at that time?
Do you not see these games from October 2023:
$ tar tjf 10_4.tar.bz2 | head -n 20
57951834.json
57951835.json
57951836.json
57951838.json
57951837.json
57951840.json
57951841.json
57951843.json
57951844.json
57951845.json
57951846.json
57951848.json
57951847.json
57951850.json
57951852.json
57951855.json
57951856.json
57951857.json
57951859.json
57951862.json
1 Like
Thanks for looking into this @siimphh. I am looking at the games based on the start_time field, and I don’t see any of the game ids that you listed. Here are the game ids I see in that time range:
game_id,start_time,start_datetime
56409831,1695850669,2023-09-28 00:37:49
56580379,1695995088,2023-09-29 16:44:48
56651534,1697084370,2023-10-12 07:19:30
56661570,1699128547,2023-11-04 22:09:07
56687876,1696931901,2023-10-10 12:58:21
56691244,1696310736,2023-10-03 08:25:36
56694900,1696685480,2023-10-07 16:31:20
56703225,1696029976,2023-09-30 02:26:16
56705090,1696534201,2023-10-05 22:30:01
56724631,1696843558,2023-10-09 12:25:58
56743905,1697341094,2023-10-15 06:38:14
56745657,1698766444,2023-10-31 17:34:04
56756714,1696842989,2023-10-09 12:16:29
56761044,1695748193,2023-09-26 20:09:53
56769430,1697319653,2023-10-15 00:40:53
56869508,1696592997,2023-10-06 14:49:57
56876218,1696514606,2023-10-05 17:03:26
56876846,1697049662,2023-10-11 21:41:02
56901414,1696851480,2023-10-09 14:38:00
56908973,1698612108,2023-10-29 22:41:48
56913973,1696888201,2023-10-10 00:50:01
56943327,1696891948,2023-10-10 01:52:28
56943748,1697240099,2023-10-14 02:34:59
56989297,1697477687,2023-10-16 20:34:47
56998269,1697253952,2023-10-14 06:25:52
57029183,1696994743,2023-10-11 06:25:43
57036116,1695796910,2023-09-27 09:41:50
57045353,1700254317,2023-11-17 22:51:57
57058005,1696901343,2023-10-10 04:29:03
57066599,1695966693,2023-09-29 08:51:33
57067484,1696683542,2023-10-07 15:59:02
57084370,1698337045,2023-10-26 19:17:25
57104592,1695775548,2023-09-27 03:45:48
57107182,1696181222,2023-10-01 20:27:02
57122375,1695996803,2023-09-29 17:13:23
57132942,1696094868,2023-09-30 20:27:48
57134110,1697627898,2023-10-18 14:18:18
57145162,1696997613,2023-10-11 07:13:33
57151136,1697634132,2023-10-18 16:02:12
57154484,1699271646,2023-11-06 13:54:06
57173855,1696007154,2023-09-29 20:05:54
57176206,1696256486,2023-10-02 17:21:26
57177892,1695883534,2023-09-28 09:45:34
57184290,1696004028,2023-09-29 19:13:48
57191368,1697558915,2023-10-17 19:08:35
57191448,1696780109,2023-10-08 18:48:29
57194975,1700402220,2023-11-19 15:57:00
57195688,1695831195,2023-09-27 19:13:15
57195693,1697487903,2023-10-16 23:25:03
57195697,1698332256,2023-10-26 17:57:36
57195706,1696825729,2023-10-09 07:28:49
57208873,1695927768,2023-09-28 22:02:48
57221040,1696145598,2023-10-01 10:33:18
57247553,1695748977,2023-09-26 20:22:57
57250103,1696106905,2023-09-30 23:48:25
57261401,1696504656,2023-10-05 14:17:36
57266187,1696562919,2023-10-06 06:28:39
57278363,1696019325,2023-09-29 23:28:45
57283501,1695761247,2023-09-26 23:47:27
57290582,1696594035,2023-10-06 15:07:15
57291278,1695901068,2023-09-28 14:37:48
57296081,1696248048,2023-10-02 15:00:48
57305396,1695739962,2023-09-26 17:52:42
57307022,1695677400,2023-09-26 00:30:00
57309942,1696359940,2023-10-03 22:05:40
57311104,1695829908,2023-09-27 18:51:48
57320521,1696510970,2023-10-05 16:02:50
57324197,1697447883,2023-10-16 12:18:03
57324943,1695677231,2023-09-26 00:27:11
57328209,1695927539,2023-09-28 21:58:59
57329575,1695770083,2023-09-27 02:14:43
57331639,1695693221,2023-09-26 04:53:41
57331645,1695773927,2023-09-27 03:18:47
57331679,1695909958,2023-09-28 17:05:58
57331747,1695823259,2023-09-27 17:00:59
57331755,1695898441,2023-09-28 13:54:01
57331814,1696413789,2023-10-04 13:03:09
57333491,1696729481,2023-10-08 04:44:41
57333495,1696733521,2023-10-08 05:52:01
57334363,1695712815,2023-09-26 10:20:15
57336098,1699464826,2023-11-08 19:33:46
57349623,1696653811,2023-10-07 07:43:31
57352740,1695758440,2023-09-26 23:00:40
57353286,1695793369,2023-09-27 08:42:49
57353475,1695836460,2023-09-27 20:41:00
57357403,1696095245,2023-09-30 20:34:05
57358773,1696061656,2023-09-30 11:14:16
57366323,1695755865,2023-09-26 22:17:45
57368624,1695861792,2023-09-28 03:43:12
57372245,1695837653,2023-09-27 21:00:53
57374567,1696617652,2023-10-06 21:40:52
57379038,1695710545,2023-09-26 09:42:25
57380643,1695677764,2023-09-26 00:36:04
57382817,1695703698,2023-09-26 07:48:18
57383830,1695993119,2023-09-29 16:11:59
57384399,1695677157,2023-09-26 00:25:57
57389497,1697605904,2023-10-18 08:11:44
57390396,1696908463,2023-10-10 06:27:43
57391505,1695701503,2023-09-26 07:11:43
57391507,1695707251,2023-09-26 08:47:31
57393342,1695782219,2023-09-27 05:36:59
Looking at the game ids instead of the dates I see that the JSON file contains 57396819 and 59088683 but none of the game ids in between.
Ok, so we definitely have games from that range saved, but sounds like they didn’t end up in the final upload.
Thanks for pinpointing the issue. That’s almost exactly where my crawl re-started (I did two crawls 2 years apart).
I’ll fix the issue next time I revisit this! I’ve made a note.
1 Like