Speaking as an allergy sufferer…
I’ve been told time and again by well-meaning obnoxious busy-bodies that if only I would quit drinking milk and consuming dairy my allergies would go away. You know, I have actually tried this. Twice I’ve eliminated all dairy for 3 months. Know what? Didn’t help. Just as much itching, sneezing, rashes, digestive upset and phlegm as before. Of course, the anti-milk crusaders don’t like to hear things like that and often flat out tell me I’m lying.
I like milk. It agrees with me. I will continue to eat it until such time as it doesn’t agree with me. On the other hand, I think the latest recommendation I heard - that an adult woman drink at least four glasses a day, in other words, at least a quart, is excessive, even if it’s skim. But that’s personal opinion YMMV.
Now, here’s the scoop - there are many ethnic groups in the world who lose the ability to digest milk at adulthood. It’s a genetic thing. This includes most (although not all - Mongolians seem to be an exception, for instance) Asians and about half the African tribes. The estimate on one of the linked websites that 95% of people of African descent are lactose-intolerant applies to African-Americans, whose ancestory derives mainly from West African. East Africans such as the Masai - who may derive nearly all their protein from the blood and milk of cattle - retain the ability to digest lactose into adulthood.
By far the largest group of people who tolerate lactose into adulthood are Europeans, with the trait being most common in Northern Europe and slightly less common the closer you get to the Mediterranean.
So… if you are genetically blessed with a certain trait milk is not harmful and indeed a useful and nutritious food for life. If you do not have this trait then you should not eat dairy. In areas where the trait is rare (Japan and China, for instance) the native cuisines will be lacking in milk products. Where the trait does exist (India, for instance) the local cooking will have milk products.
The upshot is - if you retain the ability to make lactase (which allows you to digest milk) and you do not have a food allergy to dairy then eat or don’t eat dairy as you please. If you don’t produce lactase you can either stay away from dairy or buy enzyme tablets that let you digest the lactose. If you have a food allergy don’t eat dairy, period.
Even with those who can digest milk - medical problems can arise from consuming too much of anything, including milk and calcium. The fat issue has been done to death.
The issue is another food fad type thing, with an extra push from the rabid vegetarians.