It's absolutely worth it. It's set bonus lets you ignore damage once every 30 seconds, which is one of the most powerful effects in the game and basically means defense doesn't matter. I actually used it so much in my Master Mode playthrough that I was able to kill moon lord with it, and it wasn't even close.
Also, for future reference, defense actually becomes less and less important the farther you go in the game. It's Damage Reduction that really helps you in mid-lategame, and you don't play summoner for that.