Ah now I see the problem… maybe?
That icon is not saying it does not have access 
When it doesn’t have access, it displays the symbol without any numbers in it, instead an exclamation mark. After I log out my browser from OGS:
When it has access, it knows who you are and how many games you have, and it displays them.
It’s grey because it’s boring: no games to play.
If you try it when you have some, you will see that the numbers are on a green background (IIRC, I have none myself right now!)
The little pop up that says “it has access to this site” is just telling you that the site on the current tab is the one that it is reading (as opposed, say, to the beta site).