Honestly I don't think any enemy banner items can be considered one of the rarest items since with enough tedium you can find and kill 50 of them though several enemy banners are contenders for difficult acquisitions namely the banners from rare enemies or enemies which become rare if not acquired prehardmode i.e. Angry Dandelion(hardmode), Blood eel, Dreadnautilus, Dr. Bones, Eyezor, Hemogoblin Shark, Ice bat(hardmode), Nymph, Mothron, Rune Wizard, Sea Snail, The Bride, The Groom, Tim, Undead Miner(hardmode), Undead Viking(Hardmode), Wandering Eyefish, Zombie Merman, Alternate evil mimic etc.
The true hardest to acquire items are world gen/pillage items with limited areas they can spawn in.
These are almost exclusively cosmetic with exception of the sword in a stone the two accessories from pyramid chests and the finch staff but you will probably need to generate multiple worlds to get them in the case of some like the Pyramid loot you might need to generate a lot of worlds or use a specific seed you know has them. The latter is essential for two very specific items unique to special seeds