Only loosely related since I'm on a different system, but the game didn't start for me either, on GOG's Linux version. Turns out it was a problem with their installer, amazingly enough. In the start.sh file, there is a reference to "Terraria.exe" which doesn't work so hot on a Linux system. I changed that to "Terraria" and the game then started up. They use a file called "Terraria" (without an extension, as that isn't required on Linux) that seems to do a if-else check for Mac vs Linux. When the start.sh file didn't link to this file, the game couldn't start.
I mentioned this in a bug report to GOG about the resolution being weird and the mouse not reaching the whole screen, so maybe they didn't pay attention to the start.sh issue, so if you are in touch with them, please let them know to fix this. It's pretty serious.