They will not spawn if you destroy the entire dungeon entrance.
Terraria Community Forums
Animus Viral
Leoloop
good to know
Techhunter_Talon
More specifically, don't break the blocks on level where the Old Man stands during the night. As long as there is a platform of blocks there that is about as wide as the entryway of the Dungeon then the cultists will spawn. Also, be sure there's plenty of vertical space for the sigil. If you're unsure then don't touch the Dungeon entrance at all until those cultists spawn.