I think that Challenge Links , a feature that will be released Real Soon Now, will help your situation a lot.
You will be able to create an “Invite Only” challenge, then share the link to it in your Group, and anyone in the group can accept your game.