I agree that there are people who think "summoner bad" because of the damage or unreliability of summon AI, however, I personally think it's the worst class for several other reasons:
1) Lack of variety. It doesn't help that summoner has the lowest amount of unique weapons, a total of 35, mages have twice the amount, whilst ranger and melee have four times as much weapons (a bit less if you remember a lot of them are useless ore swords/shortswords). When you also consider the fact that only a third of all those weapons are actually viable, you end up with a class that will play every single run pretty much the same, which can get stale extremely fast. I have to say, I personally adore sentries, but there are very few of them, and they are often considered to be really bad by everyone. Oh and also, accessories and armor: summoner accessories only give you minion slots and simple stats, and all unique summoner armors (except obsidian) are boring as hecc. I mean, spooky wood armor is supposed to be on the same tier as shroomite, beetle and spectre, but those three provide cool, unique bonuses, whilst spooky armor is just stats. Even stardust armor - you beat the game, you expect something awesome considering all the other armors were so mundane - and all you get is another minion.
2) No real good start. Now, I know you don't have to play pure classes, but a lot of people find such runs more fun than using whatever you can get, myself included. Melee and ranger pretty much can get their first weapon at the start, mage is a bit more difficult but they have quite a few options, whilst summoner? Pray for a finch staff, or dive deep underground for a bucket of lava and give some flinxes a warm bath. But what if the snow biome spawns behind a corruption/crimson biome? You can either waste a ton of time trying to pass, and creating the largest graveyard ever seen, or create new worlds until rng blesses you.
3) most weapons require extreme grind/rng. Finch staff, vampire frog, Sanguine, Desert tiger, Xeno, deadly spheres, Ravens, OOA sentries, Pirates, and so on - either require extreme rng to obtain, or need you to take down hard bosses/events, perhaps multiple times before you actually get the weapon. This can seriously kill the pacing and fun of the game.
So, to conclude, I think summoner is the worst not because it's actually bad - but because pretty much only people who are willing to invest into it will find it fun. Almost anyone can pick up ranger or melee, or even mage, have decent success and fun with it, but summoner has very little to offer and might throw most players off.