I’d say that black people usually don’t have this problem, either. They might be recognized as “not from around here,” but they will still be assumed to be American.
And in the various places where I’ve lived, the same is true of Latinos, as long as they have an American accent. But it wouldn’t surprise me if this is less true as you get closer to the Mexican border (where the immigrant population is higher) or further away from it (where the entire Latino population is lower, immigrant or not).