OGS Game SGF Downloader

Internet Go OGS
1

Hi everyone o/

I wrote a program called OGSGFGet that grabs any number of your latest games from OGS in SGF form. I posted this in another thread I made but I figured it would be easier to find/explain in its own thread.

You can get it for Windows here:
https://github.com/EReeves/OGSGFGet/raw/master/OGSGFGet/Binary/OGSGFGet.zip

For Linux and Mac here:
https://github.com/EReeves/OGSGFGet/raw/master/OGSGFGet.mono/Binary/OGSGFGet.tar
Note: For those of you on Linux/Mac systems that already have the python runtime installed or would prefer installing it over the mono runtime(the mono runtime is 5mb smaller \o/); scroll down a few posts to find thouis’s python downloader.

Requirements

Windows: .NET 4.5, Most of you will already have this installed, if it gives you an error you will need to install this. http://www.microsoft.com/en-au/download/details.aspx?id=30653

Mac & Linux: You will need to install the mono runtime, most if not all package managers have it (I think as “mono-runtime”) http://www.mono-project.com/

Debian based distro users(like Ubuntu) please read the troubleshooting below.

Instructions

Windows: Extract the contents of the zip folder and run the executable. Follow the prompts, it will download all of the SGF files in to the same folder as the executable.

Mac & Linux: I’m not really sure about mac but you will need to open the executable with mono depending on how your mono is set up, the sgf’s download in to the same folder just like in windows.

You can extract the files with tar(or possibly through something in your GUI), the command is “tar -xvf OGSGFGet.tar” and run them with mono either through your GUI or with “mono OGSGFGet.mono.exe”

Troubleshooting

Debian based distro users, take note. (and possibly other Linux distro users)
Sorry but the mono install is a little janky on your systems so there might be a little extra effort required.

If it’s not working, try installing “libmono-system-core4.0-cil” from your package manager “sudo apt-get libmono-system-core4.0-cil”

(This is very unlikely) If it still doesn’t run you might need to install “mono-devel” and run “mozroots --import --sync” no quotes.

Any further issues please post and let me know.

Source

8 Likes
2

[quote=“Rvzy, post:1, topic:3018”]
You can extract the files with tar(or possibly through something in your GUI),[/quote]Double-click works fine on OS X.

[quote][…] and run them with mono either through your GUI or with “mono OGSGFGet.mono.exe”[/quote]I wouldn’t know how to get a GUI for mono but executing mono OGSGFGet.mono.exe in the Terminal worked fine, thank you! (I admit being quite overwhelmed by the many options when first I ran man mono and was quite relieved when your command worked flawlessly.)

Perhaps one should note, though, that this command should be executed in the folder in which “OGSGFGet.mono.exe” resides, otherwise mono probably wouldn’t find it.

Thanks again, Tom

1 Like
3

I still use Windows XP (and do not want to change it!) and Net 4.5 can’t be installed on my system. Only Net 4.0 works. Is any way to solve this?

4

The security risks man.

I’ll get you a 4.0 build asap.

3 Likes
5

Alright, it’s targeted at 4.0, give it another download.

1 Like
6

Thank you very much! Now it’s working! :smile:

1 Like
7

For those that would prefer not to muck with Mono, I have a python downloader here:

Note that it would need some hacking to get private games correctly, but it should be able to fetch public games and reviews.

4 Likes
8

Thank you, @thouis,

two things:

  1. The name “ubderdownload” contains a superfluous “d”
  2. I get this:

[~/Documents/Go Stuff/OGS Stuff/uberdownloader] tom% python uberdownload.py 52 LALA
File “uberdownload.py”, line 5
< !DOCTYPE html > // without spaces after and before < > (code/pre tags don’t work)
^
SyntaxError: invalid syntax

(I think I prefer BBcode a LOT over MarkDown.)

1 Like
9

Awesome, I made note of it in the original post.

With a little bias. :grin:

1 Like
10

I think you need to download the .py file there, via a raw link that should be somewhere on the page. But if @Rvzy’s solution works for you, I think that’s a better choice. Mine is more a proof-of-concept / one-off (I wrote it for Uberdude to get his games, < /obvious>)

11

Wasn’t that obvious to me since you EN speaking folks seem to put “uber” in front of anything you find awesome :smiley:

But no problem, because, as said, @Rvzy’s stuff runs fine w/ Mono.

12

The error you pasted makes it look like you downloaded an HTML wrapper around the actual script.

You can try this link: https://raw.githubusercontent.com/thouis/uberdownloader/master/ubderdownload.py

But if that is what you downloaded, I’m confused.

1 Like
13

Oh. I fear you’re right. Sorry for the confusion (which, obviously, is on my side).

Now, when I run the file I downloaded this time (from your last link), it began to work … and then:

Downloading game 997333...
Downloading review 38479 of game 997333...
Traceback (most recent call last):
  File "ubderdownload.py", line 70, in <module>
    "review {} of game {}".format(r, g))
  File "ubderdownload.py", line 52, in save_sgf
    sgf = get_page_with_wait(SGF_URL)
  File "ubderdownload.py", line 14, in get_page_with_wait
    response = urllib2.urlopen(url)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 442, in error
    result = self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 629, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR
14

Does OSX have Python installed by default?

15

Yes. Version 2.7.6 here. I could upgrade to version 3 if needed. OS X is based on UNIX, stuff like Perl and Ruby, etc. is also installed by default (and, of course, Emacs, vim, etc.) … or. I understand nothing of all this, though … but I can paste a command in the terminal and I also can read a man page :slight_smile:

16

The “500: INTERNAL SERVER ERROR” is a bug that I’m pretty sure @matburt is aware of (SGFs for reviews can’t be fetched from the API), though you can work around it by loading up that review in a web browser first.

1 Like
17

Mh… I’m not sure whether review games perhaps should (and could?) be excluded from such a download? Anyway, it only d’loaded one single game SGF.

18

I just pushed a version with review downloading turned off, temporarily.

2 Likes
19

Thank you!

It took a while (I see you inserted some pause, probably to prevent the server from being upset), and it d’loaded 253 game files perfectly.

20

I should probably note I made bot games optional and made it name files with more info than 123456.sgf.

It saves games as Black vs White Result
e.g. Rvzy vs Satomi W+8.5 points.sgf

2 Likes