I am the black sheep of my family. So i take issue with the negative connotation. I am not bad, I am different. I get told this every chance some family member finds a way to say it. I am ok with it. I have learned to embrace my black sheepiness.
It’s negative but also carries a connotation of being more interesting than the family conformists. I would add that I don’t think it’s used for someone who commits serious crimes. So and so the serial killer is the black sheep of his family - you don’t say!
When the family’s awful and there’s somebody in it who isn’t, maybe it’s a flock of black sheep and the non-conforming one is a white sheep? 
Coming from a family of graziers the context is my long suit.
It’s a singularly negative connotation.
The black gene in sheep could be considered as a recessive lethal. Even if the lambs weren’t killed at marking you certainly didn’t breed from them.
Mum was an enthusiastic spinner & weaver and prevailed on Dad to keep a couple of black ewes or wethers to supply her with dark wool. There were no black ram lambs on the property.
They were not allowed to mix with the main flocks which risked cross-contamination. They were always shorn after the main flocks had been cut out. They were always kept in paddocks away from the road, the house and the boundary fences You didn’t want them mixing with your neighbors flocks.
Everybody knew we had them, 'cause all sheep breeders get them occasionally. They had acknowledged value, Dad had a prized heavy jumper from jet black wool bred, grown, shorn, carded, spun and knitted on the property.
Alas, to kill the story, being recessive, a black family (if a black ram and black ewe could be described as a family) will only have black progeny.
Maybe the white one is adopted?
Well, then it’s more of a red-headed stepchild…
I have always thought of black sheep as being very negative, usually involving criminal activity.
That would make youmoorit.
I’m also the black sheep of the family and I find it used in a more lovingly teasing manner when the term’s used. More like ‘There she is, the black sheep of the family, can’t just go and be like the rest of us, has to be all different and shit, gawd’, sort of thing. But I do know that generally it does mean someone who’s been ostracized from the family for whatever differences the family deems important. In my case, my political leanings are enough for the rest of my family to consider me the ‘black sheep’ but not anything so important that they’d ever cut me off.