I think the terms in that context are equivalent. Describing him as “father” or “babydaddy” both imply that this is not a person who the mother has an ongoing intimate relationship with. Otherwise, you would call him “husband” or “partner”. The father/babydaddy may or may not have an ongoing relationship with their child; I don’t see either term as necessarily implying one or the other.
The source of male genetic material not only doesn’t have contact with his daughter by the woman referenced in my OP. I can’t say for certain if he has any contact with his other kids. So you can call him sperm donor, baby daddy, or by his name, just don’t call him the kids’ father, because that’s more inaccurate than the other terms.
But none of those words are primarily used in a pejorative sense. “Baby daddy,” tends to be. While it could technically refer to any father who is no longer in a relationship with the mother of his child, it is primarily used to refer to a father who refuses to be part of the child’s life at all.
Urban dictionary has one neutral definition, and then several negative ones.
Now, maybe “baby daddy” is used more by black women than racists. But I know I’ve been hearing it a lot more from racists myself, and thus do not feel comfortable using it as white man, any more than I would have written that second definition at UD that uses n—a a lot.