If people can’t go outside their metaphorical door without being attacked: I don’t think it’s possible to defend the attackers and the attacked at the same time.
And if the conversation’s happening in your own social circle: sometimes you have to pick a side. Because refusing to do so has the effect of siding with the attackers.
Those situations where I must choose between defending those who would offend minorities and the minorities themselves are the situations where one side acts with malice or insincerity; in other words, where one side abandons reason and refuses to even consider opposing arguments, and preemptively declares them deceptive, and incites the people to answer such arguments against their position with violence. And as the concept of liberty is the solution to the paradox of freedom, the concept of good-faith is the solution to the paradox of tolerance.