I’m embarking on a project that requires analyzing about 110,000 .sgf files with KataGo’s analysis feature, assisted by KataWrap for batch processing. My knowledge in programming and PC hardware is quite limited, so I’d really appreciate your patience and assistance.
The aim is to create a personal database of Go strategies and moves through KataGo’s analysis. I plan to explore this data later to identify patterns. Essentially, I’m on a research mission. I don’t need sophisticated GUI software or for KataGo to operate in real-time.
I’m targeting an analysis quality that surpasses professional levels, thus I’ve adjusted the settings to about: MaxVisits = 1000-1500, numAnalysisThreads = 2, and numSearchThreadsPerAnalysisThread = 16. (I’m choosing between the 18b or 40b models)
Currently, my M2 Max Mac Studio takes more than five minutes to analyze a single file with KataGo, suggesting it could take over a year to process all the files. I aim to reduce this to under a minute per file. Therefore, I’m considering buying a desktop PC with a more robust GPU, ideally with tensor cores, within a 1,500-2,000 USD budget. (To run KataGo CUDA or TensorRT)
SO MY QUESTIONS ARE: What’s a cost-effective CPU and GPU combo for my project? For example, what kind of performance increase could I expect from using an RTX 3080 or 4080 compared to the M2 Max Chip?
Despite extensive searching, solid advice has been hard to come by. I’d be incredibly thankful if those knowledgeable in tech could recommend some viable CPU and GPU pairings that won’t break the bank. Many thanks in advance for any suggestions!
Sounds interesting! I’d love to see what comes out of this project.
I’ve spent a bit of time analysing things with KataGo and discussing on the L19 forums, but not done anything on this scale. I’m not qualified to help you with the hardware setup. But I would encourage you to be very clear about your goals.
If the aim is to play with some cool hardware and software, and have fun learning about this technology, then KataGo analysis is a good case study.
If you’re seeing this as your Secret Path to 9 Dan, you’re going to end up frustrated and disappointed. You’ll put in a lot of effort creating data that you don’t know how to interpret. You’ll improve your go a little bit along the way, but the same amount of effort spent on traditional study methods (play, review, do tsumego) will give a much bigger payoff in terms of your go rank.
Of course, another goal could be to publish your methods and results, encourage others to do likewise, and be part of a decade-long effort (or more) to figure out how AI can effectively tutor humans
Maybe also evaluate how much it would cost you to run this analysis on a cloud GPU. I think that might be possible for about 100$ a month (basing this on the cheapest google cloud spot GPUs) but I wouldn’t know how long it would take to do the processing you’re hoping to do there.
