While I do feel your idea is interesting, I personally think you would be better off using
Mumble. It uses much less bandwidth, is fairly simple to set up and the program itself is a lot lighter, with a cleaner UI (although the most important thing here to note is the fact that it uses less bandwidth, so voice lag is much less common and it won't affect user's connections as much as Teamspeak or Skype would, and the same goes for the program being lighter, causing less stress on the user's machine.)
Feel free to ask me anything you want about Mumble, I use it on a near-daily basis.