Are you using a wireless connection or a wired connection? It sounds like the connection might be dropping. visit speedtest.net and screenshot the results if you would.
Also, whoever is host should see the server command box, what does it say when you connect and when your connection drops?