How to scrape my games?

OGS Development
1

hi, is there any way to scrape my completed games? i have tried jsoup, but can’t find any links to my games.

thanks

2

Edit: on second thought I think you mean mass download of all your games? I have never seen a way but I would imagine the API would allow it somehow?
Sorry, I can’t help.

3

I don’t think he means “scrape” as in “delete” I think he means “scrape” as in “collect all the data”.

Brute force way is to start from your profile: https://online-go.com/player/179/ and scrape the game history table by paging through it.

However, there’s almost certainly an API that will have the results on it - someone will likely be along to tell you about that soon.

(Wow, player #179, that’s old-time!!)

4

Is there a program that someone has written to download the games in SGF format (assuming that’s what you are looking for)? I guess - IMHO - the question isn’t clear enough for me to answer. Sorry I couldn’t be of more assistance.

5

@rtayek OSR (https://openstudyroom.org/) has a bot scraping OGS games. I guess best way is to ask devs at their discord, they will know the most.

1 Like
6

OGS game records:
https://online-go.com/api/v1/players/{player_id}/games?ordering=-ended&ended__isnull=false

It’s a json where "results" is a list of games.

To download a sgf for a specific game:
https://online-go.com/api/v1/games/{game_id}/sgf

For a more detailed game record:
https://online-go.com/api/v1/games/{game_id}

If you elaborate what you want to do in more detail, I can probably help more.

PS: To crawl games on OGS puts a high load on the OGS servers. After about 50 games, you will get throttled. The server answers then with a 429 and a json telling you, you got throttled. You then have to wait awhile before the next request will work.

4 Likes
7

the problem is that our teacher (mr, yang) reviews our completed games. so i need to click on profile and scroll a lot. hitting back in the browser, takes me to the top of the profile. so lats more scrolling. and it’s hard to remember which game it is.

i tried walking the dom with jsoup, but i can’t find the links to my games.

thanks

8

this is great! it has each game id. i should be able to scrape this and fool around with out games. mostly, i just want to find those that we do not have a review for.

thanks

9

looks like it’s paginated json. not quite sure how to deal with that, but it looks like there are lots of libraries i can use from java: How to parse JSON in Java - Stack Overflow

thanks

10

There is a script on GitHub that allows you to download all your games in sgf format.

3 Likes
11

i managed to get some stuff to work - java code using json simple and some shell scripts.

thanks

12

Interesting thread. I never intended to dld all my games, but I assumed that I can do it anytime. (It seems that I cannot. )

Hmm, the solutions gave till now did not pointed to a real solution, nor the ones I assumed will work.

Where is this bot? I joined OSR to test the bot, did not found it.

This could be a feature for site supporters. The games can be found on your profile anytime, but if you want to have a local database, organized better, of course you may want them to be downloaded automatically, not to save them one by one. So there should be a setting (available for supporters eventually) for saving locally your game after is finished. This will save the server of the load… I assume.

It allows you only if you know how to use it. I am not totally helpless around computers, but I did not managed to let the script to allow me to put some load on the OGS server.

So these are the methods offered here. My solutions are also three.
-First is the simple save complete html page with the browser, which should save the sgfs too, at least those on the opened page.
-Second is to use an download manager. I tried three, none worked.
-Third is to use an mirroring software. I always used WinHttrack with satisfactory results. I saved entire sites with it, and failed only I had to provide password, and few other rare cases with complex software like Yahoogroups.

None of my methods worked here, but I did not tried too hard. It may work with different settings, but I am not interested in wasting too much time with this subject. I just wanted to share my little knowledge on the matter.

13

Hmmm, I should have elaborated, sorry. OSR has a custom league system that gathers games from both KGS and OGS (last I chekced, maybe they added more since). The bot is not a freely avilable tool to use, but is an underlying system for the whole leagues and automatically gathers games that were played by OSR members and hashtagged with #OSR in chat - to make sure it was intended as a league game.

My reasoning was that it would probably be best to ask someone who is versed in these things (Flovo or OSR) instead of going trial and error, not that there is a freely avilable tool.

1 Like
14

I see from further down the thread that you solved your problem, but I wanted to address this particular situation in case anyone else has this problem but isn’t able or inclined to wrestle with scripts and json: if, from your list of completed games, you open a game in a new tab or window (in Chrome, that’s done through ctrl-click in Windows or command-click on Mac) your scroll position in the original tab won’t be disturbed. This might even be an easier solution for you if you only need to grab a few games from your history at a time.

15

that’s a useful tip.
thanks

17

how do i get request?

D:\ray\dev\conradapps\scrapogsgames>node getgames.js 179
internal/modules/cjs/loader.js:596
throw err;
^

Error: Cannot find module ‘request’
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:594:15)
at Function.Module._load (internal/modules/cjs/loader.js:520:25)
at Module.require (internal/modules/cjs/loader.js:650:17)
at require (internal/modules/cjs/helpers.js:20:18)
at Object. (D:\ray\dev\conradapps\scrapogsgames\getgames.js:1:77)

at Module._compile (internal/modules/cjs/loader.js:702:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
at Module.load (internal/modules/cjs/loader.js:612:32)
at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
at Function.Module._load (internal/modules/cjs/loader.js:543:3)
18

This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.