Item: Dirt and Stone
Can't they leave me alone? I just wanna mine!
Mobs: All types of bats, every enemy that spawns in the underworld, and every single pillar enemy that doesn't come from the Stardust Pillar.
Most hated Mobs: Cave bat, fire imp, and Corites
I don't know if I would have an item that I consider annoying, since even annoying "objects" and blocks have neutral utility at worst once dislodged and moved into your inventory. I could consider clutter as agitating though. Lesser Healing Potions and enemy vanity drops become particularly obnoxious even not that long into pre-Hardmode.
Most annoying enemy is Tomb Crawler, no competition. It's far too common for how difficult of an enemy it is to handle, especially with its Worm AI being even more hellish to deal with in a confined deathtrap of a biome. I wish Relogic had reduced its spawnrate immensely in a previous update or something.