Restarting Livereload server fails when it is in use

When I load some page with livereload, e.g. by using the 'show in browser via livereload' and then try to restart the livereload server, the server stops and won't start again.
This is the error message:

Cannot start LiveReload server because port 35729 is already in use.

Trying to start the server then will fail with the same error. It seems though there is some timeout after which the port is released and the server can start again. Still, it somewhat defeats the purpose of the 'Restart' function.