Focusing on whites vs. blacks for a moment:
When X is negatively affected, it’s pretty hard to repair the damage done to X without directly addressing X. By this I mean entire races have been disadvantaged in various ways, and it’s going to be hard to close the gap without giving them advantages that won’t also benefit the privileged.
That’s the crux of the issue. If you help the disadvantaged, the advantaged cry foul and want the “better deal too.” If you try to help everyone, it doesn’t actually solve the issue and the gap still exists.
Whites, for instance, have always gotten more – but they think if another group gets more, that means they’ll get less. They feel it’s a zero-sum situation where someone’s going to lose, and they’d rather it not be them. People are extremely uncomfortable with the idea that they technically got preferences they didn’t work hard for.
“It’s not my fault slavery happened – they should just get over it.” Even though it didn’t occur while we’re alive now, whites still benefit from the ripple effect over time. I mean, free labor for 200 years… benefiting whom, exactly? Same thing with indentured servants who came here from Asian countries.
How does a group of people move on from being dehumanized? Working 20+ hour days, enduring atrocities and not being paid anything, not being allowed to live, and not being allowed to read or get educated… then suddenly you “free” them. How on earth do they catch up?
AA, as stated earlier, technically helps white women the most if you look at the numbers. It’s the same sort of thing with desegregation back in the day. It was supposed to help, but instead, many black schools were penalized – they had to use their money to recruit and diversify their student body whereas white schools didn’t have to do a thing. The end result was that many black schools had a better quality of education pre-desegregation.
The answer to your question, IMO, is that there is no truly “fair” way to do it. Many problems are solved with a higher quality of education + cost reduction, so I think a lot of the focus has to go there. However, it’s worth asking if it’s really “unfair” to give benefits to someone who has been disadvantaged for the sake of your gain.
It’s not an easy issue – but I think if people are going to discuss race honestly, they need to be critical about the history and not naively handwave away crucial issues. Check out Jared Diamond’s Gums, Germs, and Steel sometime for more on socioeconomic evolution.