They need to be considered to be a member of your "party" for you to warp to them, which is inherently a consensual agreement. You would only be able to use this to steal from your friends and allies, which is quite a rude thing to do! Wormhole is still a fantastic choice for other reasons, though, I can't deny invisibility's sheer utility, either; its downside of disappearing when you get hurt is much less of a concern, too, since humans tend to get hurt much less often than a Terrarian does.