It’s also a common mistake to confuse “not a good speaker” with “dull-witted” and “unimaginative”. 
Why should there be panelists asking questions? Over here, we’ve just the two candidates arguing with each other, and they’re more likely than any panelist to ask the most embarassing questions to their challenger. And of course, they can’t know the questions in advance. It usually make for a very interesting debate, with sometimes great moments. I wouldn’t want a panelist messing with that.
Interesting question, clairobscur! I am not sure why the U.S. debates have evolved to this format. It is unfortunate…It would be nice to go to a format more along those lines (with some sort of moderator there to at least insure one candidate doesn’t monopolize all the time). [I believe that in the 2000 race, the Gore campaign did want more give-and-take than the Bush campaign wanted in the debate format, although I am not sure Gore was advocating a complete one-on-one sort of debate.]
Two reasons I can think of - one, the media probably doesn’t want a [figuratively speaking] knock down, bare-knuckle fight because it doesn’t look very “presidential” and two, one or both of the candidates probably would refuse to do the debate.
To make it easier to lob pre-screened softball questions to the candidates, of course.
I’d also argue that it’s a common mistake to confuse “not a good speaker” with “a quality that a successful president doesn’t really need”.