That depends, when do you fight the duke? If I get Cosmic Car Key, then he's not so bad, but without it, he definitely would be worst barring Moon Lord.
Expert Skeletron is Satan, but I'm not sure if he's actually worse than Duke Fishron or expert Plantera.