I’ve noticed a divide between people who grew up Christian, culturally Christian, or secular but without a specific religious-ethnic identity (which is to say, they identified mostly with the dominant white Anglo-Protestant culture without necessarily being conscious of it), and people who grew up Jewish, whether practicing or secular.
Christians IME draw a fairly sharp mental divide between the secular and religious aspects of Christmas, even if they don’t think about it much. There might be some overlap–maybe Santa shows up at church after the service to entertain the kids, or an atheist might enjoy a religious song or two, but even those are seen as something of an exception, a sop to the other side or a willingness to embrace both sides, but for most people inside the “majority” (not for long!) American culture (British too, I think), it really is, as some people said, two separate but simultaneous holidays, and we all know more or less what things are part of one or the other holiday. Atheists who grew up in that culture are fine celebrating secular Christmas, and may get offended (I do) when told that their celebration doesn’t exist or is “really” Christian.
Jewish people, however (and I would expect the same goes for members of other minority religions and cultures that don’t celebrate Christmas) see Christmas from the outside, and don’t make those distinctions. They see it as a single huge cultural meme practiced by the dominant Christian culture and distinct from and alternative to their own cultural and holiday traditions.
As an atheist and former Christian who grew up without any particularly strong Jewish identity (though my father is Jewish) I have no problem picking out the secular aspects of Christmas and celebrating them in a way that neither I nor any Christian would mistake for a religious celebration. But while it is possible to have a purely secular Christmas, it probably ISN’T possible to have a religiously neutral Christmas that doesn’t feel like an imposition on Jews (and, as I said, probably Hindus and Muslims, though I won’t speak for them).
In fact, Judaism can be interpreted as forbidding Christmas celebrations for practicing Jews just as the Jehovah’s Witness faith prohibits it for its members. In that context, whether the celebration is secular or not doesn’t really matter. It’s not universal.