Do you see a lightning bolt icon in their time box? If so, then they have disconnected from the game (perhaps due to a poor connection or perhaps by intentionally leaving). The timer is there to cause the player to always timeout within 5 minutes even if there is more time left on their clock (they timeout even sooner if they had less than 5 minutes on their clock).
Here are some other threads discussing this feature.