the only one I have played was infinite, but from playing burial at sea, I can tell that I would enjoy the first game WAY more than infinite, simply from the atmosphere alone.
Columbia is neat, but honestly, it isn't presented as well as rapture. even though rapture is confined to large underwater skyscrapers and domes, it somehow feels a lot more vast, and way more daunting. almost more like a fortified castle than a city. it gives the impression of a stronghold, while Columbia just feels like an ordinary town that happens to be floating.
the residence of columbia seem to simply be zealous and over righteous.
the residence of rapture are horribly disfigured and driven insane and dependent by supernatural substances.
gee which one sounds like a more threatening town to be in?
as for the real freaks of the show
I do like the bioshock infinite enemy design more than the other games. specifically I like creepy robotic type monsters, so my favorite enemies in infinite are the Handymen and the Motorized Patriots. that's not to say I dislike the enemy design in the first games, in fact I think they are excellent, but through personal bias I like Infinite's more. by that I mean, Infinite demonstrates exactly my favorite enemy design.
I also like the weapon design more in infinite. I love early 1900s guns. although I find it funny that the machine gun was actually a german SMG and was made in an otherwise highly patriotic racist, does not like other nations town. however, given where the plot ends up going, it's actually a great attention to detail that Columbian hardware borrows design from foreign and sometimes even unusually advanced engineering. (Handymen as an example)
all and all, I prefer the first two games for the atmosphere, and infinite for the character and weapon design.