Yes, it's called a Page File, and computers do automatically do it. It stores less-commonly-used items, since storing all currently in use data there would probably wear down the HDD or SSD very quick due to the constant writing and reading, as well as being far slower, to the point where it wouldn't be useful.
But yes, the logistical problem is that Terraria, as a 32 bit program, is only capable of using 4GB of RAM, period. No ifs, ands, or buts about it. It's an inescapable limit without rewriting the entire game to be 64 bit.
The Linux and Mac versions of Terraria supposedly have 64 bit options, but the devs have stated that bringing 64 bit to Windows is "not a priority", so who knows if that will ever happen, much less if it will be any time soon. I assume tModLoader, 64 bit or not, would need to match the version of Terraria you have installed. Perhaps not; that would be nice.
That said, I wouldn't reccommend attempting to make such a mod. Many people have found that adding such an insane amount of content would do more harm than good. You would end up (metaphorically) with every concievable combination of every weapon type, damage amont, projectile, etc. The spritework would have to suffer thanks to the sheer number of sprites needed, the bosses, enemies, and weapons would likely end up being copypasted because giving even half of them unique behaviour would take years of work from a full-time paid dev team, much less one or a few modders. It would be an impossible mess to balance or bugfix.
It's not feasible on any level. Most of the mods that have even 300+ items, 50+ enemies, etc. suffer from these problems to an extent, even after years of additions and alterations and balance patches and updates, and many mods are being rewritten with a lot of content intentionally removed because it was unbalanced, repetetive, or pointless. Terraria is a huge game, and adding even more stuff gets overcomplicated fast.
If you do want to make a mod, try quality over quantity. Add unique things, make each item feel like a different item, integrate them seamlessly, make the bosses feel unlike the vanilla bosses, make the enemies new and exciting. More content does not make something better; good content makes something better.