Using FME to retrieve data from OGS and Tableau to display them

Lys · June 20, 2018, 2:20pm

Hi there!

This is a very very newbie topic!

I would like to retrieve some data from OGS in order to make some chart. For now I would be happy to get my data (games, players involved, who won, board size and so on).

Thanks to @S_Alexander I got the very basic of it:

I can ask the server using a http GET request (never heard of before)
Answer can be in json format (never heard of before)
In order to do this I have to send a sort of URL to the server (wow, sounds easy)
Example of this URL can be found on the apiary (well, some seems broken, see later)
They can be cheched using any browser (this is really easy)
Answers can easily become HUGE
So be careful not to overload server
It could be necessary some sort of authentication to state that you’re not trying to hack or crash the server

Here is an example:

I went to the apiary looking for something easy to check. Found this “conciste-player-list” (BTW, is that english?)
https://ogs.docs.apiary.io/#reference/players/list-players/conciste-player-list
I copied the link
and tried it in my browser: “page not found” (here it is: seems broken)
then I added “api” in the url and it worked
https://online-go.com/api/v1/players/
Answer is made of 54376 pages

Ok, leave this for later. Let’s find something easier!
A single game.
Apiary URL still misses the “api” keyword, but now I know.
immagine

So here’s the data:
https://online-go.com/api/v1/games/13240061

Now for the big question: how can I submit this request other than with my browser?
Some uses python, but I don’t.
I’m trying to use FME, which is a software I use at work.
It’s very nice and user friendly and is specific for retrieving, modifying and storing data.
I never used before the FME “HTTP Caller” but I can learn. So I did and it worked!

Now I know how to:

Send an http GET request
flatten the json
extract attributes that I want to
store them in any database format that I like

So why am I here, still asking for help?
Well, I’m a kind of sorcerer’s apprentice who can easily make disasters, so I would be careful.
Also, I think my story can be useful to others.
But let’s focus on the “careful” thing!

I didn’t use authentication 'til now. Should I?
If I should: I didn’t understand the “token” thing, what’s the “Developer portal” and so on
https://ogs.docs.apiary.io/#reference/authentication/request-a-user-token
what can I do not to overload the server? How many data can I retrieve without doing harm (and possibly being kicked off)?
The 54376 pages of the player’s list, for instance, will it be too much?
Also, is it splitted in to pages only for the browser or also via direct GET request?

I dare not to just try because I don’t want too many brooms flooding my cavern.
Can anybody help me?

Lys · June 19, 2018, 9:20am

Well, actually at this moment I could very very easily:

use a single HTTPCaller to ask for the player’s list
redirect its output to another HTTPCaller that could ask for the history of each player
redirect it to another HTTPCaller that could dump all data of each game ever played on OGS

Now do you see what I mean when I talk about disasters?
I have a huge gun in my hands and I don’t want to harm anybody (also myself).

Eugene · June 19, 2018, 9:39am

Not that I’m aware of

Eugene · June 19, 2018, 12:23pm

I think it is a great question to be asking. You recognise the power of that rope you are holding

If the server is properly designed, you should not be able to hurt OGS with a finite number of one-at-a-time requests.

So if you iterate through every player and all their games, asking for the next after you received the previous one, and you stop after that (don’t go on indefinitely) a well designed server should be able to cope. Maybe it slows down a little while that’s happening, but it ends.

What you must not do is go on indefinitely asking for stuff, and even more importantly, you mustn’t ask for lots of stuff in parallel.

For example, if your environment (python - I know what that is - or FME - I don’t know what that is) allows you to make many requests at the same time or quickly in succession before the first one finishes then that’s nasty behaviour. Your numerous requests can swamp the server: this is how “denial of service” attacks work.

If you are asking for a finite amount of stuff, one thing at a time, I would have thought you should be OK.

That being said, it would certainly be the polite thing to do to ask @anoek directly before you decide to ask for “all the data”, even one request at a time.

GaJ

Lys · June 19, 2018, 3:15pm

For now I’m asking for my games, 100 at a time to reduce the number of requests:

https://online-go.com/api/v1/players/308797/games/?page_size=100

But my requests start immediately one after the previous because it’s a loop. So, it’s 4 requests in few seconds.

If the page_size was lover (say 10), that would be 40 requests in few seconds.

I don’t know which case is better for the server.

Also, I’m not so clever, so I need many trials and errors to become at the end. So today I asked for my data many many times (and still don’t have them!!!).

Everything without authentication…
I feel very dumb now.

crocrobot · June 19, 2018, 3:18pm

if you go to the api root in a browser, you can see the various paths through the api, it is useful for finding what is available and the proper path for the request.

BronzeV · June 19, 2018, 4:39pm

Don’t beat yourself up over it. As long as you don’t ask for anything obviously huge (I don’t know if it’s possible but asking for the complete list of players in 1 page would be bad), and as long as you keep the amount of requests per minute to a reasonable number, you should be fine.

Eugene · June 19, 2018, 11:13pm

I think being careful is a massive great first step.

It is also actually the responsibility of the server to protect itself against silly single requests. So if the server can’t cope with being asked for the complete list of players in one page, it simply shouldn’t honour that request. Typically that’s done with a maximum page size.

The biggest thing to be careful about is being sure that your system is not making many requests at once.

If it genuinely waits for one request to finish before sending the next one, and is not in an infinite loop, then that’s pretty well behaved.

Some software will be able to make lots of requests, wait for them all to return, and assemble the result. Don’t do that to the OGS server

Another good guide is “don’t do anything that takes a long time”. If it takes a long time, you’re probably making the server work hard, and that means other people might feel it.

The good news is that unlike Mickey, you can pull the power plug on all those brooms at once if it starts to go wrong. The symptom of go wrong is “takes a long time”.

GaJ

Vsotvep · June 20, 2018, 12:00am

I was going to make a pedantic response about infinity but I feel that’s not really helping anyone… I’ll just sadly weep about the irrelevance of set theorists to daily life problems instead.

Also, consiste: think that’s a typo

Lys · June 20, 2018, 7:47am

Hi everybody, many thanks for your support.

I proudly announce that I did it!

I throttled requests so to make one every 15sec.
I made a loop that uses the “next” keyword, so to be able to download all data, whatever is the lenght of the page.
I downloaded summary data of all of my games and I’m about doing some stats on it.

Now I have just a big question mark…

In my profile page I can see:

11 ongoning games
my history made of 38 pages of 10 games each + 7 games in the last page

It sums up to 398
My downloaded games are 400.
So, should I try to find the two ghost games?

((( BTW, G0t Stats says that my total games played on OGS are 371 )))

Lys · June 20, 2018, 10:51pm

Here is my first attempt making some chart about my data:

As you can see, correspondence games can take very long time to finish.

I used Tableau software.
The dashboard on my PC is interactive: I can choose the board size from top right filter and then charts will update, showing just a portion of data.

It’s possible also to publish dashboards online, over Tableau Public portal, so that everybody can navigate and interact with them.

Lys · July 15, 2020, 8:49am

It’s more than two years since last update, but I have a new project and it’s specifically about this thread.

The “Through the years” tournament has started and I’d like to create some visualization about it.

My intentions:

using FME to retrieve tournament data periodically
using the site data.world to share these data
using Tableau to make some viz and post them on Tableau Public

If I manage to complete that, I should be able to update data very easily and everybody could see them and interact with them.
Sounds nice.

Lys · October 4, 2020, 8:25am

using FME to retrieve tournament data periodically CHECK
using the site data.world to share these data NOPE
using Tableau to make some viz CHECK
post them on Tableau Public NOPE

There’s still much to do.
Now I’m still exploring data and trying to create some interesting chart.
Points 2 and 4 are completely new to me.
Point 2 requires that I learn some PostGreSQL.
I could try point 4 though, and see what happens.