The destroyer is always hard, especially if you're like me and play classes/accessories with low Defense. The most consistent way I can find to kill him (in Master mode, at least) is to build a small box a decent way away from structure (as he can climb vertical walls/stray blocks) but below space (wyverns = no fun in a boss fight). You should be able to comfortably jump for height and strafe left/right about 20 blocks. With a couple 2-4 block platforms in the bottom of the solid box (in the corners is your best bet) rain down fire and when he unleashes 300 probes at you, take a break and focus them down, using the box for safety from the lasers and dodging them as they charge you. Then alternate shooting destoyer/probes. If you're still struggling, build a respawn point in/on top of the box and a nurse housing. Nimbus rod, summons, golden shower (for debuffing) and anything that pierces (or shatters) is your best bet. High-cost mana weapons are not your friend, especially not in boss fights. Harp's bouncing is not bad, especially if you aim down at a slight angle.
tl;dr: Build a box; alternate attacking destroyer and his probes; use piercing weapons.