Well, if you saw some of the horror stories people have about them at the cons, you'd know.
(People dropping full diapers in the lobby of a hotel...)
But personally, there are stupid people everywhere, so its not fair to judge someone based on their preferences like that.
That, and they liken the yiffy part of babyfur to pedophilia, like non-furs liken regular yiff to bestiality/zoophillia.