No.
Say I’m Jewish, and someone finds this out, and drunkenly starts following me around at the house party making Nazi salutes. I am not an asshole for asking him to stop and apologize, even if I do so loudly enough that I’m heard over the music/stop the music so he can hear me. Because the presence of a Nazi (or an asshole who uses Nazism as trolling), that needs to be addressed by everyone (see my sigline). Everyone prioritizing maintaining a fake calm over dealing with an issue like a Nazi presence is the assole. Sunlight is the best disinfectant.
Same-same for other racists, or people using racist expressions and not apologizing when called on it.
Fun ≯ safety