I’m enjoying this discussion - however, I’m a bit concerned that we are focusing so much on the TREES that we are losing track of the FOREST - i.e. the possibility of getting a functional game review queue up and running
Is there a way to GUARANTEE that every single peer-to-peer review will be of the finest possible quality?
NO - and IMHO - this is one of those situations where “the perfect is the enemy of the good” - in other words, I believe that it might be more valuable to just get a peer-to-peer review system up and running with A FEW functional guidelines / suggestions, allow it to hit the ground running, and then adjust as friction points come up - rather than discuss this to the Nth degree and never actually decide on what to build.
As such, I suggest ideas that will SUMMARIZE the suggestions posted above with the aim of finding a middle ground that most can agree on, and then do a live test with actual humans, see how well that works, and adjust from there.
Examples: for SDK / DDK games, volunteer reviewers can only review games from players that are 8 kyu weaker than them. Dan reviewers need only be 1 dan stronger than the player they review (i.e. a 4 dan player can only get reviews from 5 dan and up, etc).
A given review should (in aggregate) be no shorter than 800-1000 words (about 1.5 to 2 pages of printed text in 12 point font on a blank page - with the understanding that these will be spread out over many comments across various important points throughout the game. Conversely, try to limit your reviews to under 5000 words, as this might overwhelm the reader.
My 2 cents.