I"m trying to download all the games I’ve played on OGS. I’ve trying using the python script uberdownloader which works but after a few downloads I get the following error:
Traceback (most recent call last):
File “ubderdownload.py”, line 81, in
“game {}”.format(g))
File “ubderdownload.py”, line 67, in save_sgf
sgf = get_page_with_wait(SGF_URL)
File “ubderdownload.py”, line 22, in get_page_with_wait
response = urlopen(url)
File “/usr/lib/python2.7/urllib2.py”, line 154, in urlopen
return opener.open(url, data, timeout)
File “/usr/lib/python2.7/urllib2.py”, line 435, in open
response = meth(req, response)
File “/usr/lib/python2.7/urllib2.py”, line 548, in http_response
’http’, request, response, code, msg, hdrs)
File “/usr/lib/python2.7/urllib2.py”, line 473, in error
return self._call_chain(*args)
File “/usr/lib/python2.7/urllib2.py”, line 407, in _call_chain
result = func(*args)
File “/usr/lib/python2.7/urllib2.py”, line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: FORBIDDEN
403 is a strange response, I would’ve expected a 429, because it sounds like you’re getting rate limited. Try increasing the delay between requests? SGF downloads can be pretty taxing on the server.
Also possible there’s some config issue preventing you from downloading one of the games, and it fails when it hits that one. Do you know if this only grabs completed game SGFs? Maybe it’s trying to grab one from a game in progress with analysis disabled or something. Can you tell which one it fails on?
EDIT: It looks like you talked about this in chat and it’s because it’s hitting a private game. Can you link the script you’re using? I suspect the modification is super minor to check for private games. About to take a look at what the API gives back about this.
EDIT 2: Yup, public/private isn’t exposed in the API. I’m wondering if the script is using your account or someone else’s, and if it’s yours, if API is blocked form private games you should be allowed to see. Regardless, it wouldn’t be hard to surround this in a try-catch to continue on HTTP error 403 as well.
I’m also familiar with this script. It doesn’t currently support authentication to allow access to private games. It should handle this case gracefully. I will create a fix and Pull Request to address this. It might take me a day or two.
It will download a game in progress, but it will skip it from then on.
The script also will reduce the rate in response to a 429.
Once I have some cycles again, I plan to make this script more robust.
I also have a script that will rename the files based on the players and date in the file.
Already fixing it in a forked repo. I’m going to test it on my account real quick (although I don’t have any private games). The issue was that the try-catch loop uses “raise”, which dumps a stack trace and exits instead of gracefully handling it as you would expect a try-catch to do.
Evangelizing here, but I would also recommend maybe looking into requests instead of urllib2. It handles this a lot better, and makes for cleaner code.
Thanks for the reply. I was able to get the majority of my games. The error 403 refers to private games. This experience has made me more interested in the OGS API
