It's a case of RAM unfortunately. If I remember correctly, Terraria only allows up to 4GB of RAM at its maximum, your pc specs means nothing towards this limit, it's how the game's engine is (allocating more ram won't work unfortunateley...)
Calamity & Overhaul are most likely the two mods that do use the most RAM of any given mods of Terraria while the game is running, so having both Overhaul & Calamity running while creating a world (which is one of the most ram-expensive interaction of the game, because items might "break" during the process while Overhaul keeps the location's data of EVERYTHING and won't remove what's on ground), the world won't create at all.