Nexus 5:
The perfect option for the vanilla Android experience, just like your Galaxy Nexus used to be for the last 2 years.
You're free to root it and install custom ROMs and everything. The experience should be very similar to that of your Galaxy Nexus.
It should also be running very well due to that it has the most optimizations. Expect to get all the Android updates first. It's also the only Android device to date that ships with version 4.4 KitKat (and should be the first to get whatever the next version will be).
The price of it also makes it probably the best deal you'll get of any phone right now.
LG G2:
Hardware-wise it's a great phone, however LG is known to lag behind with the software. Unlike the Nexus 5, LG is in charge of making sure it stays up to date. The latest Android version 4.4 may find its way to the G2 in a few months.
It has some nice features not seen in the Nexus 5 (or other phones as well) but at the cost of taking a few months longer than certain devices to get certain system updates.
Padphone Infinity:
It has the padphone concept which is pretty cool, although it ships with Android 4.2 which is now over a year old. It's said to be getting updated at some point next year.
Galaxy S4:
To be honest, Samsung does make some great hardware on the inside (although the outside is just a cheap plastic feeling). That said, and I am fairly biased, their UI is horrible. It can be easily addressed by installing some 3rd party apps like Nova launcher but I usually prefer when things are easy and not too laggy and dumbed down out of the box.
As with almost all phones, it has some uinique features which, in my opinion, honestly is mostly just some niche stuff in order to stand out. It also comes fairly bloated with a lot of pre-installed apps you might not care about too much. It's not too big of a deal as they can be removed/disabled but, just another thing that might be quite annoying until' you get it all set up correctly.
HTC One:
Fastest updates of all non-nexus/google edition models it seems with the 4.4 update is just around the corner, set for release shortly after christmas or early January. The unlocked "Developer edition" model has already gotten the update.
The built-in speakers are the best of any phone on the market and build quality is similarly the best of any phone in this list, using a solid aluminum construction as compared to the plastic on all the other devices. It really 'feels' more premium in the hand compared to the other devices I've tried.
The camera ain't awesome but it does hold the crown for low light photography and is adequate for being a phone camera.
My personal pick would be the Nexus 5 or HTC One. I am bit biased though since I do myself own the HTC One. I have however converted it to the "Google Edition" version because I want the Nexus-experience.
If you can hold out a few months more, there should be new models early next year replacing both the Galaxy S4 and HTC One (thus dropping the price of these further if you don't need the latest).
I wouldn't say specs between any of these modern devices makes much of a difference.
Of course the higher-end Snapdragon 800 chip is indeed a bit faster and all that, but the Snapdragon 600 isn't too far behind. They are all very fast and smooth overall and regardless which you choose it'll surely be an improvement over the Galaxy Nexus (I had a Galaxy Nexus myself for 2 years until' this summer when I finally decided I needed an upgrade).