This is a common misunderstanding. Public defenders are often the most experienced criminal attorneys in the courtrooms. They handle thousands of cases, they know the law, the process and the judges better than most private criminal attorneys. Just because you pay money to someone for representation doesn’t mean you’re getting the best representation.
As @Procrustus says, a judge can call on any attorney in their jurisdiction to represent a criminal defendant. Public defenders’ offices are set up with primary public defenders, then conflict public defenders, then conflict-conflict public defenders and so on. When we ran out of names on the conflict-conflict-conflict list, we started calling members of the local bar and wheedling them into taking the case. The attorneys know the judge wouldn’t call on them if they had a different option, so for the reasons stated by @Procrustus, the attorneys always agree to represent the troublesome defendant. Into each life some rain must fall.
I can tell you our private lists did include some “do not call” attorneys because they were viewed as not up to complex or high profile cases. We never called anyone we felt would be a detriment to the defendant or to the process. The last thing you want in an important trial is some inexperienced lawyer screwing up the case.