Why are the Trump defectors having such little impact on the polls and Trump's chances?

Many of the replies are very enlightening, but are focused on Trump’s base. As difficult as they may be for a liberal like me to understand, I was trying to get at a different issue. That being, of those who did support Trump in 2016, were the vast majority really his base?

My assumption was that there are meaningful numbers of people in other groups who could be pulled away. Based on the polls and Nate Silver’s numbers, though, it seems I may have been mistaken in those beliefs. Are “moderate Republicans”, for lack of a better term, really that rare of a beast, that they no longer have any influence, even as a tipping point demographic?