Thankfully I managed to find a workaround for that. If you put your headset on, you can slightly lift the glasses by temple tips so it's still on your ears, but it doesn't get in the way of headphones and doesn't angle too much.
Terraria Community Forums
Bezixx