There is a similar problem with tournament invitations. If I send an invitation to Fred and he closes the notification without accepting or declining, there is no way for him to recall the invitation. Also I cannot reinvite. I get a message similar to the one you showed:
If both people are online at the same time then it is possible to do a workaround by changing the tournament to ‘open’ and then back to ‘invite only’ after they have joined.
This problem has significance for any tournament like Ladder Legend where people qualify for an invitation.
The simplest solution I can think of would be to remove the small (x) closing option from notifications that have a Yes/No option.