Oh boy, another class thread? Time to spread the good news about summoners!
Well, first of all, what are you doing using "just minions"?
If you aren't using other weapons alongside your minions, then you've missed the point of the summoner class. Minions leave your hands free, so you can and should use other items. If you don't, then you're just crippling yourself arbitrarily. It's like if a mage decided they were only going to use accessories that affected magic damage and mana usage, meaning they couldn't use items like Celestial Shell, Destroyer Emblem, or even wings because they aren't strictly "magic". It's an arbitrary, unnecessary limitation and not the way the class was intended to be played. You're supposed to use everything available to you, so short of a self-imposed challenge run, you should.
(Incidentally, this also can go the other way, meaning that if you aren't using summons with your melee, ranger, and mage characters, then you're doing them wrong, too. Always use every legitimate advantage you can get.)
Now once you've gotten over your silly misconceptions about the summoner class, you'll find that they can actually be extremely powerful if you play them right, especially at endgame. In fact, as someone who has played every class including summoner extensively and enjoys tinkering with them to try to bring out their full potential, while most people will tell you mages have the highest single-target DPS in Terraria, I'm here to tell you that's not true. It's summoners. With a 10-segment Stardust Dragon and an SDMG loaded with Chlorophyte Bullets, my summoner can reach a combined total of around 10k max DPS (2-3k from SDMG, 6-7k from Stardust Dragon). Meanwhile, my mage's Last Prism struggles to reach 7-8k, and when inflicted with mana sickness it drops even lower.
Of course, that's only at endgame, and it's true that Stardust Dragon has a huge DPS leap over the rest of the summons. Not to mention mages still beat summoners in multi-target DPS, since Last Prism's beam can be way longer than a Stardust Dragon and summoners can't use piercing weapons alongside the Dragon. But summoners are by no means weak, even before you get the Stardust Dragon. Not only that, but even if their effectiveness at boss fights is debatable, they're undeniably far and away the best class for exploring, building, mining, farming, etc., basically every other part of the game.
That isn't to say they're perfect, obviously, since they have that low defense, but the low defense is there for a reason. If they didn't have it, then they would completely overshadow the other classes. Summoner has been my favorite class ever since it was added, and it's only continued to find ways to make me fall in love with it.