Eh, I'm not really sure about this one.
The biggest problem I see with a change like this is that, items like the Hercules Beetle will see much less use. When I was a young Summoner in 1.2, I regularly wore my Hercules Beetle along with the Papyrus Scarab, not knowing how summon damage calculation worked, not knowing that my minions were living projectiles. It was a waste of a slot, so I was a bit slow and frail.
This would also leave players at a decision on whether they should keep Tiki armor for the extra minion and defense, or use Spooky for higher damage at the cost of both. Going up against the literal Celestial Pillars while lacking some damage or defense depending, will surely snap a Summoner in half unless they are an excellent builder for those encounters. However, currently, a smart Summoner can prepare better with the Spooky-Tiki swapping method (or using the aforementioned Old One sets with or without the Forbidden Treads).
Another thing I take into account is that Summoner was nonexistent when the game was around for its first two or three years. I don't really have a problem with the whole complicated swappy-loadout deal because it adds to the fact that Summoner is quite unique compared to the other classes (and I also feel pretty advanced doing it). Calamity actually made this change to Summoners, but I'm not really a fan of it. This is mostly a personal preference, so not everyone will feel the same way. I like putting more thought into my loadout, personally, especially when the results pay off well in the end. It's a rewarding feeling to go through all the effort of building your damage up.
However, I would be more open to this idea if a few things changed:
Items like the Pygmy Necklace could have a Tinker upgrade, perhaps combining with the Papyrus Scarab and some Stardust Fragments. Maybe that sounds a bit too OP, but perhaps it would still grant just 1 extra minion but with slightly higher damage. This way, I wouldn't need to always worry about whatever damage I'm lacking since I already have a decent compromise (and I can put something other than a Hercules Beetle in my vanity accessory slot).
Forbidden is very solid of an armor on its own, and Stardust is a straight upgrade to Tiki and Spooky. The only suggestion I could make to armors is that perhaps Tiki could get a slight bump in damage, as it's currenly as strong as Spider armor but with a bit more defense and an extra minion; an upgrade I personally don't mind, but that's mostly because I can use the swappy-loadout to make up for what I somewhat lack. Considering I get the Hercules Beetle and the Pygmy Necklace at the same time as the Tiki armor, I definitely feel like I've gained a huge and much needed step forward as a Summoner.
We could also get minions to deal critical hits, but they would probably have to be quite a low chance considering how fast and hard some minions can hit already. We already have the Lightning Aura and Forbidden Sign sandstorm already, and those perform quite well in some circumstances. There's a chance that we'll actually see more minion crits in 1.4 and I'm looking forward to that.
I'd also be in favor of possible unique weapon modifiers for summon staves. They could keep all the same magic weapon reforges, but seeing modifiers for the minions themselves would be pretty nice, such as their movement speed, attack speed, and field of vision. This way, they could make up for their lost damage in other useful ways.
Which leads to my final thing; a possible active weapon that deals summon damage. What I mean is, a weapon that deals summon damage without necessarily summoning minions or sentries. Imagine a tweaked Bat Scepter that dealt summon damage, or a weapon that spawns minions that immediately blow themselves up upon hitting a wall or enemy. The special dagger weapon from Thorium comes to mind, which spawns a temporary soul to aid you with each enemy you kill with it. Perhaps these "active" weapons would be the ones to deal critical hits, and would give more power to those who play Summoners as strictly "summon damage only" (I don't mind using subweapons of other classes to aid my minions' damage though).
I also understand the dangers of making Summoners too powerful if you add more active ways to deal damage on top of the constant area DPS of their minions. To that, I suggest, make minions update their damage values properly ;P
I think I've come full circle now. Phew.