Perhaps pushing for clarity on the question of “racist or not racist” misses a more useful point, and tends to suggest a binary choice between “people using the word that way are inherently jerks” and “we should all go ahead and use the word that way”.
It’s great that English tends to adopt words from other languages, but I can see it might be a bit problematic that native users of the language borrowed from might sense a misfit between how English speakers use the word and how native users do. It might kind of trivialize and misrepresent the word. It’s a kind of cultural appropriation. If we combine that with the choice to keep using the word in a discussion involving specifically those native users, it could be insulting, right?