Yes, “swarthy” was the word I thought of as soon as I saw the thread title. Also remember that a coomon trick in the late 1800’s / early 1900’s was that very light African-Americans tried to pass as “swarthy” Hispanics to escape full racist treatment. (I think too it was a big deal that Lucille Ball was married to a hispanic and used that in her TV show - certainly ground-breaking for the time…)
Along with all the reasons mentioned above, add mass migration. Whether Irish, Italians, Chinese, etc. - they arrive in large numbers in a short time, not knowing the language, and so tended to congregate in big city ghettos where they had a support structure of people who spoke the same language and were willing to offer help. But to the outsiders who refused to help them, they appeared a monolithic group who did not want to integrate and did not associate with others, thus aggravating the racism and xenophobia that contributed to discrimination.
Canada saw the same thing in places like Toronto, which starting in the 1960’s saw a massive influx of Italians. While the situation was not quite as bad as say, 1900 NYC, many of these immigrants were less educated and ended up working in the local construction boom. I remember at least one Italian student in University in the 1970’s complaining that his high school guidance counsellor said he should consider construction, not university as a career.