Human review queue feature?

[tonybe]

I have a very different idea as to how to encourage this culture of game reviews:

• create a place on OGS / OGF where users can get into a QUEUE to request that their game be reviewed, and announce it so that everyone is aware
• gather a pool of dedicated volunteers that are naturally motivated to provide those reviews because they enjoy participating in that review culture
• create a minimal set of hoops for those volunteer reviewers to jump through to show that they are capable of providing decent game reviews (show previous reviews?), and at what rank levels (DDK, SDK, dan+ etc)
• provide some standard checkpoints for what each review should include to encourage (but not demand) some level of standardization
• once the system stabilizes, offer a running estimate of how long (on average) a person will need to wait in the queue before their individual game is reviewed.

This way - a new-ish player could request a review of a single game without committing to something like a tournament.

thoughts? questions? suggestions?

PS - BONUS ROUND - if this system gets up and running, perhaps we could apply the same model to people asking for TEACHING GAMES at various rank levels.

[garlicsoup]

Great idea!

@tonybe

• users can get into a QUEUE

Personally I’m very enthusiastic about giving and getting reviews, so I might even get into a QUEUE to do so, but I might fear that many people would view such a format as a bit of a chore, whereas the tournament would not only attract more people but probably result in better reviews.

[JethOrensin]

@tonybe

• create a place on OGS / OGF where users can get into a QUEUE to request that their game be reviewed, and announce it so that everyone is aware

This is probably the best idea, imho.

If you do not mind, to add a bit of financial practicality to it, since creating such a side in OGS would undercut its business model for subscriptions which is the AI-reviews, you could make this a tiered item.

A queue where people would be encouraged by a pop-up at the end of the game to “request a review from higher ranked players” and once there, they could get into the PREMIUM queue where OGS could promote that it will be providing reviews by high dan amateurs or low rank professionals, with the cooperation of one of the teaching academies for Go that have such teachers.

It could then create a self-rotating funnel from OGS to reviews to lessons and back to OGS where the players will have the ability to have more feedback and meet prospective teachers and academies or enroll to tournaments and what-not.

As a starting point you could have each OGS subscriber have a “high-dan review request” per month, so that those higher placed reviewers don’t get overwhelmed with too many requests. Also the uniqueness of the review per user will make each user cherish their review and want to request one only if the game is “very good/interesting” for them and, potentially, for the reviewers as well, who could also gather more material for their classes, books, tutorials or whatever else they are into.

[tonybe]

Yes! These are all great ideas!

In general I’ve been trying to figure out ways that OGS / OGF can provide “Mentorship as a Service” - for lack of a better word.

As things currently stand, the only way that something as huge as OGS can continue to offer a free-to-play tier is through the work of many volunteers - and here I’m talking about MODERATORS. They’re doing MODERATOR work - resolving conflicts, annulling games, banning troll players, deciding whether X incidents count as “escaping” or “stalling” or not, etc.

Then there are all the various NATURAL HELPERS and GO ENTHUSIASTS - like me - who are not organized in any way - offering reviews, teaching games - or for DDK players - doing more of a hand-holding / guided tour because all they’ve ever played is live 9x9 and concepts like territory vs influence or direction of play are completely foreign. These volunteers are offering free services because the very act of passing this information on is mutually enjoyable to both parties.

However, in order to find and access those HELPERS - it takes a bit of bravery for a DDK to raise their hand in an open forum, and request help. I wondered in a previous thread how we can better foster and retain these players - because it’s exactly what we need if we want the player-base of Go to grow in the West - retain those people who arrive here all excited, and then realize that reaching 1 dan may take many years.

How do we connect beginners to helpers? How do we simplify that process?

How can we make sure that both parties actually take this seriously, and show up for their session, and not flake out?

We’re aiming for the process to be mutually rewarding, so that people keep coming back from both sides, and the program grows naturally - i.e. more SDK and dan players may be attracted to the idea of offering the occasional review if they’ve heard positive stuff about the program from others and want to try it out.

And yes, I think adding this to the monetization of OGS is a grea idea, but obviously, you’d need a dedicated team of dan players to pull this off, so I’ll leave that to the mods to hash out.

But yeah, I was thinking something for the free-to-play tier, because I can think of a lot of reasons that a new-to-the-game / new-to-OGS DDK might be too shy to ask for help, or too meek in their abilities to join a tournament. How can we give them a button to click that says

“Would you like some help from other players on OGS? CLICK HERE”

My 2 cents

[bagagwa]

Maybe it could be called something like the Go Teaching Ladder!

[AdamR]

I have split the topic into a new thread as I believe it deserves its own.

---

I must admit I very much like the idea, and think it deserves further discussion. It would be a great thing to support community engagement, and make Go more accessible, and inviting to new players. And it mostly builds on tools we already have, so it might be relatively realistic even as development time goes…

Things to note or possibly learn from. It is very much similar to what https://gokibitz.com/ tried/tries to do.

• It is an interesting idea to have multiple people commenting on a single game. Probably more likely to get at least “something”, but the quality of such reviews is dubious imho, as the reviewer often had only a cursory look, and cannot offer any “general summaries” (as in you often play gote, you often leave weak groups…)

My only major worry is that it is hard to predict supply and demand of the requests, and it would be very disappointing, pointless, and counter productive to have beginners asking for reviews, and getting none. So it might be a good idea to set some limits initially?

[jlt]

The length of the queue could be limited.

[Jon_Ko]

Maybe limits to enter the queue could be established. I.e. one player could be prohibited to have more than X games in the queue.

[shinuito]

I’ll cross post this suggestion also as it’s related

@shinuito

Maybe as a stripped down version of tonybe’s suggestion, one could imagine since we have automated reviews of games (with at least the top 3 biggest mistakes being highlighted), one could opt in to submitting those key moves for review to a review queue (one or more).

Then as a player you wouldn’t be on the hook for a whole game review, but instead you’d try to review a key mistake from a game.

It would also almost function like a puzzle queue for players reviewing since they’d be looking through real game mistakes.

[tonybe]

Full disclosure: in suggesting both the Human Review Queue and the Teaching Games Request Queue, I was quietly trying to come up with ideas for volunteer opportunities that I would be motivated to commit actual time and effort towards (wink smiley)

I think another functional element that would be needed to make this smoother is that it could be broken down into rank tiers, so DDK games, SDK games, and Dan games would be in different queues, and be handled by different groups of reviewers.

As such, the expectations for the type of info that someone provides for a Dan game and a DDK game would be quantitatively and qualitatively different, and both sides (the requester and the reviewer) would come into the process with some of those expectations already set.

Comments that may make perfect sense in an SDK review (“your move here was aji keshi - and it would have been better to go here for direction of play” etc) will probably leave a DDK user completely confused - because they may not be aware of those concepts yet. As such, there would be an expectation that - if a volunteer was taking on DDK game reviews / teaching games - that there would be a bit more hand-holding and walking the beginner through basic concepts rather than just saying “this move would have been better” and expecting them to understand WHY it was better.

Likewise, SDK players would not be providing Dan level reviews - I know that’s probably obvious, but I wanted to point out that the rank differential cuts in both directions.

And yes, I think in terms of both quality and quantity - it would probably be a good idea to set the expectation on both sides that a review should take up X amount of words, or take Y amount of time to generate - because otherwise some volunteers will spend 10 minutes, and other will spend 3 hours, and you’ll have reviews varying wilding in both quality and quantity across reviewers.

TL:DR

Rather than just throwing up a Human Review Queue and saying “Go For It!” - it would probably be good to generate some GUIDELINES and BOUNDARIES for people on both sides of that queue before they started, so that they would go into it with certain reasonable / sustainable expectations.

[ronin3]

@tonybe

some volunteers will spend 10 minutes, and other will spend 3 hours, and you’ll have reviews varying wilding in both quality and quantity across reviewers.

While I understand the concern, not entirely sure I would value amateur effort at a flat-rate. 10 inspired minutes might be well worth 35 minutes putting in a lot of effort exploring unproductive lines. Wouldn’t it be simpler to vote on the reviews so that high-quality reviews (regardless of the time spent) “bubble-sort” themselves to the top?

[JethOrensin]

@AdamR

Things to note or possibly learn from. It is very much similar to what https://gokibitz.com/ tried/tries to do.

True, but not many people even know of its existence and you have to download the sgf from the Go server you played it and re-upload it there. Here this process would stay within OGS, so it would have much more traction.

@AdamR

• It is an interesting idea to have multiple people commenting on a single game

There could be tiers or different lists, like the main tournament ladders (thus those lists could be shorted by player rank, as @tonybe suggests).
List A - Games that have not been reviewed yet.
List B - Games that have had one review
List C - Games that have had more than one reviews
List D - Games from Site supporters in need of review.

So, if you want to provide a review and “List A” is empty you can go ahead and review a game from “List B”. “List C” could be the unofficial “Games that have been reviewed” list, if someone wants to have a look at them and learn from them or even add another review if they so like.

@AdamR

So it might be a good idea to set some limits initially?

Let’s say you can apply for “1 review per 7 days” or something like that?
After all getting too many reviews at the same time means that you won’t have time to study the feedback, which is a very reasonable explanation for anyone that complains that they’d like more reviews per day.

@tonybe

And yes, I think in terms of both quality and quantity - it would probably be a good idea to set the expectation on both sides that a review should take up X amount of words, or take Y amount of time to generate - because otherwise some volunteers will spend 10 minutes, and other will spend 3 hours, and you’ll have reviews varying wilding in both quality and quantity across reviewers.

I am not sure you can set/limit that though… some games might not have enough friction or things to point out or someone might be less detailed in how they review a game.

Getting a review is better than not getting anything. I think that’s a sensible first step.
Making quality assurance promises (that the review is also going to be good) now turns a volunteer that wants to help out into a worker that has to provide a quality service.

That will eventually lead to bickering (“the review I got was bad, the review my friend got was cool”) and then comparisons (“I want this reviewer not that one”) and then behaviour that will probably need the moderators to come along and swing the banhammer towards the most vocal grumblers.

Raising the expectations too much, might bring this down before it begins.

@ronin3

Wouldn’t it be simpler to vote on the reviews so that high-quality reviews (regardless of the time spent) “bubble-sort” themselves to the top?

That’s also very good and you could change the shorting of the review list by date, ranking or rating.

[fuseki3]

It’s a nice idea !

I think another functional element that would be needed to make this smoother is that it could be broken down into rank tiers, so DDK games, SDK games, and Dan games would be in different queues, and be handled by different groups of reviewers

Maybe the requested reviews can bé sorted into potential rank bands, or have ‘‘sort by rank’’ /‘‘sort by recency’’

Dan players would probably bé happy to review DDK, SDK and other dan player games, for example – I recall learning a lot from a few, rare dan player reviews when briefly DDK and SDK and have nice memories of thé nice players who took thé time to review thé games or teach me a few basic concepts such as shapes / connections.

(And nice memories of watching other DDK or SDK receive reviews from kind other players many years ago, when it was common on KGS)

It would bé nice to bé able to provide that type of nice envoronment for beginners or newer players playing, too.

[fuseki3]

On thé other hand, to prévent reviews requested outpacing available reviewers, it’s probably important to make very visible to all site users something liké,

''Would you liké to help other players learn & review their games ?

Click here to join the OGS volunteer game reviews’’

And also, something visible to players in games, in addition, such as

‘‘Submit up to 2 games per week for review.’’

(2 or 3 9x9 games reviewed per week seems a better pace than only 1 9*9 game per week, for improvement and enrouragement, although 1 19x19 game per week seems reasonable also.)

Should review numbers bé limited differently by board size ?

Should a player’s reviews requested be grouped or displayed together, to help teach better, especially in 9 * 9 or 13*13 where it is easier to understand what a player may bé confused with or which areas may bé helpful to teach, after seeing a couple of their games ?

(Especially for DDK / SDK, in m’y expérience of having reviews requested of me)

(For example, reviewing even another dan player’s 9x9 game, with its intricacies, is much, much faster for me than reviewing one of their 19x19 games – I can usually review 4-6+ 9x9s of any particular player, in thé time of reviewing a 19x19, and 2-3 9x9s in thé time of reviewing a 13*13,

And usually 2-3 13x13s in thé time I’d review a 19x19.game.

9*9 seems much simpler to review than 13x13 or 19x19, to me.

It seems to make sense also to encourage, or ask explicitly of players, submitting at least 2-3 9x9s for review, if a player is wishing to improve and submitting 9*9s, to have a clearer idea of their understandings and areas for improvement–I often find it is clearer with at least 2-3, if 9x9 games – or 1-2+ games in general if 13x13 or 19x19

[tonybe]

Re: the idea that each review request could / would receive more than 1 review

Funny, I just naturally assumed that we would have more people requesting reviews than we would have reviewers available, especially in the 15 - 5 kyu range where we have the peak of our player-base Bell Curve, and the cohort probably most likely to request reviews in the first place

I have not checked out GoKibitz (largely because I don’t casually visit sites that require a login) - but yeah, I’m thinking we would be better off building our own model based on the current culture here - that seems more realistic.

Assuming there would be more demand than supply, I’m guessing that just getting 1 review would be a matter of waiting in the queue for a few days or so - as such - I’d be surprised if we could offer a model that offered multiple reviews per game, since there would be fresh un-reviewed games waiting for their first response.

Re: standardization of review length / efforts - I didn’t want to focus TOO much on this - more just to float the idea that we’d want to suggest some general guidelines as to what constituted too much (i.e. 30 pages) or too little (1 paragraph) - the goal being to make the system practically useful to the requestors and functionally sustainable for the reviewers in the long run.

[ronin3]

If the guidelines can be succinct and take up 2 to 3 sentences tops, then sure why not.

If they become anything resembling the current “community moderator” guide, I wouldn’t want that driving away potential reviewers. Just my two cents

[tonybe]

@ronin3

If the guidelines can be succinct and take up 2 to 3 sentences tops, then sure why not.

I completely agree - short, sweet, and to the point would work great - just hit the 3-5 main points rather than trying to come up with answers for every possible eventuality and edge-case, etc.