I’m a very good driver.
Comically, this point about the subjective experience of driver and passenger is something I try to bring up to convince them they should not complain.
This does not work for obvious reasons.
As to noticing a difference–that’s another thing. When I am a passenger, I do not pay attention to the things the driver is doing. Because I’m not driving. I don’t register a judgment in my mind about whether I feel safe or not. I’m just… sitting.
About close calls as a measure, I’d be curious to know how to be sure whether an incident really was a close call or not. That was the exact point of contention in the latest tiff we had over my driving. She was convinced I nearly ran into a car, I was convinced I didn’t. We both had access to the same external facts, but from her point of view those facts add up to a close call, while to me, they don’t. This could be due to my subjective knowledge of what I was doing, or this could be me fooling myself. I think it’s the former, but am not certain.
Learning to drive in a way that feels safer to a passenger, though, should obviate the whole close call question so I’ll focus on that. I just hope it’s actually learnable…
If more than one person feels unsafe when you drive, it might be worth checking into a safety course. Get a professional involved to evaluate your driving. Then you can improve or else tell your passengers to stick it.
Yes, I’ve been looking into this today. It seems like everything is videos and online courses. I am having trouble finding things that involve an actual person actually sitting in the car with you 
That’s what a wife is for!
Dad lets me drive slow on the driveway every Saturday.
Fortunately car accidents are fairly rare. You can be a pretty bad driver and not have that many. One of my sons is a bad driver. Nobody can tell him that, very sensitive. But it’s not that the rest of the family is ‘more conservative’. We just don’t tail gate people the way he does, we keep two hands on the wheel (he even makes 90 deg turns with one had, instant gteed disaster? no, good driving habit? no), we don’t talk non hands free on the phone (or even often at all)etc. IME lot of younger male crap drivers just think they know what they are doing so don’t have to have good driving habits, but there are some fairly objective standards for good habits, and they don’t have them.
In every internet discussion about driving some people, usually younger guys not only, will admit to being the ones weaving through significant traffic at way over the speed limit (not the case with my son, fortunately). If you do that, you’re a shitty driver. There’s no way skill will entirely overcome the risks that poses. So you might be a skilled but shitty driver in that case. Other bad drivers are just unskilled in ‘stick and rudder’ terms or lack of situational awareness or both. There’s some kind of Tolstoy type rule there, not each and every bad driver is bad in a unique way maybe, but many different bad drivers are in many different ways. Genuinely good drivers are all fairly conservative (on public roads), at least pretty good on situational awareness and good on physical driving skills per se.
Can you ask your passengers to more precisely describe what feels dangerous about your driving? That might be really useful.
I was once in a car with a co-worker who many people had described as a bad driver, but she, like you, claimed that she was always hyper-aware of her surroundings. She was terrible. When she asked me about it afterwards, I was able to point out several specific things that she was doing unsafely and why they felt okay to her but dangerous to others. In her case, one of the problems was that she had a tendency to drift across the lane (within her lane) while she fixed her hair, took a sip of soda, closed the bottle again, put it away, and then corrected the drift a microsecond before she went into oncoming traffic. She felt safe because she knew that she was about to put her hands on the wheel again, but her passengers were terrified because they didn’t know when she was going to stop dicking around and not cause a head-on collision. It is possible that someone can articulate what specific actions you are doing that feel unsafe.
Also, keep in mind that many things that feel unsafe to passengers are, in fact, unsafe. As an example, fast/sudden braking will get you to a stop 99% of the time, but you’re depending on the brakes performing well, the road traction being good, no distractions, etc. Good drivers know that 1% of the time, you’ll hit an oil patch or the rear end of your car will pull right or something, so they slow gradually to give room for error. The driver above was only drifting within her lane, which should be okay, but we all know that someday she is going to spill her soda while trying to take a sip and the microsecond distraction will take her over the edge. Leaving a buffer for the unexpected is important.
It’s possible that you are both a marginally better driver alone and a terrible driver in general. If more than one person said my driving made them feel unsafe, I wouldn’t kid myself that I was actually a good driver.
Another thing to consider is that whatever you’re doing that’s unexpected and frightening for your passengers is probably unexpected and at least slightly distracting for other drivers.
Take the woman mischievous mentioned above. I’m not likely to have an accident because someone else is drifting within their own lane, but it is going to attract my attention for at least a fraction of a second. That’s minor, but it’s not good. And if I think there is a chance she’ll actually drift into MY lane, it could be a major distraction or even cause me to react unnecessarily, maybe even serving into the lane on the other side.
Likewise, if I see someone tearing up behind me like they’re about to ram me, I’m likely to stay focused on my rearview mirror for longer than I otherwise would. No big deal (other than being scary) if I’m at a complete stop, but if I’ve started moving even slightly when I notice, it could cause me to rear-end the car ahead of me. Technically it would be my fault, but still…