You can’t afford to abandon a sizeable fraction of your base. These “far-right loonies” supported Perry, Bachmann, Cain, et al in your primaries, when given a choice.
You should abandon them, of course, and you will, eventually, but it takes a lot of guts to stand up to ridicule by Rush Limbaugh and Donald Trump and Ann Coulter. You don’t have that kind of fortitude now.