I have Gog version of Terraria and was able to install game launcher(along with tAPI) properly.
Do you select the Terraria folder when installing? If you do that, it shows an error because it searches for Terraria.exe in (example) C:\Program Files\Terraria\Terraria(it auto-fills with a second Terraria folder)(But I'm not sure if it's GL or Tapi that does that)
Check if this is the problem.
If it complains about it having to be Steam, try creating folders in C:\Steam\SteamApps\common\Terraria and put a copy of Terraria.exe inside. It was a work-around for when GL didn't work with GOG version.