Alright, after failing to decompile on my 4GB ram laptop, and an 8GB desktop, I was finally able to decompile on a machine with 64GB of ram. The biggest load I saw was about 18GB in use, so I bet a 16GB ram machine with a typical page file would make it through the process. This is just an FYI for anyone thinking of contributing to the mod source code.
Anyway, problem: Decompile works, but applydiff.exe "has stopped working". A bunch of them pop up at once. This was with the "setup" button. Does the setup button do "decompile" and all 3 patch commands? I tried the patch buttons themselves but the same errors happen. Also, I don't really get the "diff merged" and "patch merged" buttons. What exactly are they doing?
Here is an example from the log
"patching file 'src/merged\Terraria\WorldGen.cs'
Assertion failed: hunk, file patch-2.5.4/patch.c, line 343
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information."
I tried find replace all "\r\n" with "\n" as someone else alluded to, but it didn't work. Any idea?