I love scary game monsters, and Terraria has some of my favorites.
Face monsters. My first trip into the crimson was at night, and between the music and the strange noises coming out of the dark I was incredibly freaked out. Then the shrieking face monsters started coming at my underpowered little self. They had terrifying mouths on them, and they killed me before I could escape. It was awesome!
All the spiders. Due to a traumatic bite experience when I was little, I have a problem with spiders. The spiders in Terraria hit all the arachnophobe nightmares. They swarm, they jump at you, they live in dark places... Killing spiders in Terraria is pure catharsis. My heart races, my hands shake, and there is a satisfying sense of vengeance.
Finally, I find the Eye of Cthulu wonderfully disturbing. It's a flying eyeball trailing ganglia and halfway through the fight the front of the eye splits open into a gigantic tooth-filled maw. That's better than anything I've seen in horror films in ... ever! It's creepy and creative, and there is a reason that the transformed eye is such an iconic image.