Could I suggest adding a condition to it, then? Perhaps needing some form of prepared food in your inventory (soup, ale, sake, pad thai, etc...) before he'll spawn? While building houses doesn't take much time, the Merchant gives a lot of things that are very helpful early on even without the accessory mod, and needing to build a minimum of three houses (assuming you haven't met the conditions for the Dye Trader or anyone else) is a bit of a pain. Really, just in general, though, it's a bit of a pain (at least in my opinion), since all the default NPCs but the Guide have conditions that you can deliberately avoid to control who spawns, and he doesn't.