The reason why is Duke is the type of boss who demands you play by the right rules. Every single fight up to this one (talking Normal Mode) can be made relatively trivial with the right preparations. The winning strategy there is something anyone can do; drink ironskin and regen potions, get Well Fed, stand by campfire and heart lantern, and wear whatever gives the most defense. In most cases, you pretty much can't die. Even if you don't go through all these lengths, the point is, these bosses are pretty easily beatable without "learning" them through sheer stats and strategy.
Duke throws that out the window. It's still POSSIBLE to blunder to a victory, but much more advanced and specific in the prepwork needed. He's pretty much designed from the ground up to murder a player who coasted through the rest of the game without fully learning HOW to dodge. I find it interesting that a lot of the same strategy in dodging Eye of Cthulhu works for Duke, just on a much larger scale. Difference was, it was pretty easy to beat EoC while getting hit every second there is, not the case for Duke.