Most annoying item is easily the Treasure Magnet. This thing that barely has a role as a convenience being twice as likely to appear as any loot you’d actually use in a Hell chest. I’ve gotten as many as eight in one world. There’s a lot of other useless filler items that can replace good loot, but Treasure Magnet takes the cake.
Most annoying enemy is hard. There’s a lot of enemies that are unfair like Necromancer or Corite, but if I had to choose one enemy that is the most consistently irritating, it’s Lava Slimes. I hate whoever came up with making them drop lava in Expert Mode, and grief your campfires, drown NPS, and halt building.