Merriam-Webster: rejected or cast out by society
American Heritage: one that has been excluded from a society or system.
I don’t think it’s a particularly felicitous choice. It’s not a word I would have personally chosen. But I also don’t think it’s horribly wide of the mark. Those children who were stolen were literally cast out from where they belonged.
And a key point here is that the “agency” of outcasts doesn’t come into play, which was one of the repeated complaints. Innocents can be outcasts. Even if the word is wrong, it’s not so far afield as to automatically imply guilt among those who were stripped away from their homes and families.
That’s perfectly reasonable, of course.
But to repeat, what it “usually means” still does not automatically imply guilt. “Outcast” includes those wrongfully cast out.
More broadly: Precision in word choice is a high hurdle to demand of other human beings, most especially when precision in word interpretation is also imperfect. When you get down to it, basically no-one speaks or reads well off the cuff.