Yes, people do have to drive on surface streets to get to off road sites. However, in my experience those people are few and far between. I know several people who own SUVs and they never come close to a dirt road. These drivers often use them for getting to work, most less than 10 miles, the grocery store or just general driving around town.
The people like this I know claim that they are safer (in reality they are not) and they like sitting higher up in traffic. To me SUVs are extremely inefficient and wasteful.
Also, most of the people I know who have them are single and rarely have more than one other passenger. And when they commute to work, they are the only occupant.
In my opinion they suffer from the American (primarily US) syndrome of if it is bigger it is better and it is my god given right yadda yadda yadda.
If you do live or drive off road, then fine, but if you live in the city and never use it for what it is designed for, it is like killing a fly with an uzi. It may get the job done, but it is definitely not the best way. That is why I sit back and smile when gas prices go up. If people have to have them, then let them pay for the privilidge. I will just use my bike more often.