It drives me nuts, too. Unlike some people, I can drive and talk on my cell phone without becoming a traffic hazard.
I have a theory about this. Usually when people are on a phone, the parts of their brain not dealing with the phone go into auto-pilot. You might play with the phone cord, walk around the house aimlessly, doodle, etc. It becomes habit. This is usually fine, except when you happen to be doing something that requires your attention, like driving. If you habitually go into auto-pilot when on the phone you have to have the sense to break that habit if you’re driving. I gather from observation that not many people do.
Personally, I may get distracted by traffic and miss part of a conversation while driving and cell-phoning, but it’s never the other way around. It’s like they teach in flight training: fly the plane first, everything else second, talking on the radio last.