We (me and the group I'm playing with) have this problem as well. I'm not sure if it's supposed to be happening (can't imagine this is new intended behaviour in 1.4), but bosses just don't want to leave.
It doesn't matter if players are far away from one another, or if everyone logs out at the same time: The boss will be there going after us.
Example: Queen Bee that kills one player, then targets another player who is on the other side of the map. It doesn't matter where you are. The Destroyer followed me down to hell even though I wasn't even on the surface when it spawned. We all logged out since we were gruesomely underprepared, expected it to despawn, but nope.
I eventually grabbed another character and removed the Destroyer that way. Restarting the server might have been an option, but I'm currently renting one and it's a hassle.
Really wonky stuff.
I think bosses might disappear after everyone is dead at the same time, but I'm honestly not sure.