It seems to me like if you have a friend in your game, anyone on their friends list who has Terraria can join in, too, whether or not they're on your friends list or you actually know them at all. Instead of a "just friends" multiplayer, we ended up with a "friends of friends of friends of friends" thing, and I have no idea how far along the chain it could go. I only have one friend on XB1 who has Terraria, and people on his friends list have popped in randomly. Happened during a Moon Lord kill once... Boss spawned on the intruder instead of us, right at my main base since I built at world spawn, and just wrecked my NPCs.
@Loki, could you please see if there's a fix coming for this?