You should start from scratch, since your geared-up character would have a very easy time with 90% of the game. (Actually, with end-game gear, only the Moon Lord will remain a challenge, even with Expert Mode.)
I think it's worth it. Bosses drop loot bags that have items in them that you haven't seen before, even though not every item is that useful.
But most of all, you get to feel the satisfaction of having played through Expert Mode (if you make it, that is).
It's not for everyone though. The game will be, not surprisingly, a LOT harder. Bosses will have new moves, regular enemies will have some surprises too, here and there.