Pre-hardmode: King Slime. By the time you actually have the materials required to craft a Slime Crown, you've either already beat the Eye or have gold/silver equipment.
Hardmode: Skeletron Prime, if fought last. If you fight him first he's rather tough, and in that case the easiest for me would be the Twins.
Personally, I find the Destroyer to be incredibly difficult without piercing weapons (which are hard to come by as a Ranger, who needs to kill him to get the Megashark, which makes fighting him much easier.) As a Fighter, you do a lot of damage to him and have lots of defense, but I think that's offset by the fact that you have to be so damn close to him that it makes dodging lasers nearly impossible, and makes dodging him and his probes slightly less difficult. It's far easier for mages because of the Golden Shower/Nimbus Rod/Magic Daggers.