We have almost 40,000,000 game IDs (though not all of them are games), and aren’t really set up for what you ask, which would be a huge project and I suspect not a trivial server drain to support such a large database performing mirror and rotation searches on millions of games…
I would also like to explore what’s the common josekis and openings on OGS. I always have the feeling that there are different cultures in different servers, and players just favor some over the others.
It could be something done entirely off of OGS. Waltheri has already built up the technology to do so (is the code behind that site open source?) and the SGF dump is publicly available. It’s only matter of putting the two together. However, maybe scaling up the processing up the size of the OGS game collection is far from trivial.
I’m not sure why (by implication) the original request has to be deemed “unachievable” just because it is more ambitious than waltheris.
The fact that it would likely call for more server resources or more “whatever” doesn’t rule it out: AI analysis of our games was a big project called for more servers, and this was added because it was a good feature.
The real question then is “Why is this a compelling feature to have? And how compelling is it?”
When the latter question was asked about rengo, we found the answer was “suprisingly, this feature is compelling enough for someone to make it happen”. Similarly, an OGS Joseki Explorer was a big project calling for (a small increment) more compute, and it happened.
So we should have our minds open to this possibility…
anoek said something about the games not just sitting around in storage to be called at any time, but that they are generated on request each time.
I don’t exactly understand what that means or how it works… But I suppose it has something to do with saving space might be a speed bump for getting something like this off the ground, especially if you want it to periodically keep up with modern games and not simply stay static at a certain point.
The pattern search index files Kombilo generates are orders of magnitude larger than the input sgfs. In order to provide efficient pattern search your need to create far larger data structures than just the game tree itself. It also takes bloody ages to generate. So the resource demands of this feature seem pretty huge. I’d replace boatload with super tanker. Or maybe super-star-destroyer-ful.
Maybe OGS should invent a crypocurrency and doing this processing can be the proof of work computational task Might as well warm the planet doing something slightly useful.
Maybe I don’t understand what you’re saying, but I don’t think this is true. An efficient structure to hold this in would just be a tree, with each node holding a coordinate and # of games that hit that branch. It’s big (40M x avg number of moves), but it’s not bigger than the SGFs themselves.
Implementation would require thought, but it’s not a computational absurdity by any stretch.
Hah, waltheri is the new kid on the block, and sure is convenient for online searching of pro games away from my home PC. But it’s a smaller database than GoGoD and doesn’t have MY games. Kombilo (or crazy man Frank de Groot’s Moyo Go Studio) is how you do offline pattern search, and against your own sgf library. Which is why I want bulk download of my OGS games because then when I am reminded “Oh, this position is like that OGS game I played 12 years ago” I could whack it in kombilo and find the game. Back when I was teaching Go I would often use my OGS games as illustrative examples of concepts for students.