If I recall correctly, SGF is open-ended, and one can use new tags for extra information. That gives one way of adding the times of moves; the other would be to use specially formatted comments, such as I think I have seen Lizzie use to save extra information about analysis by KataGo or Leela Zero.
Personally I think it is well worth defining a new specific tag for, ideally in a new version of the SGF specification for Go. I would then favour a human-readable absolute time format such as yyyy-mm-dd-hh:mm:ss.xxxxxxx, where x is the decimal fraction of the second as accurate as the implementation feels like (bots might use sub-milliseconds). One could allow unchanged leading parts of the time to be omitted, and the time zone could be specified in the first tag. One could also add some notation for the remaining time control for the player who moved, or use a separate tag for that.