That was the case when I was a kid, too. People who lived outside city limits were covered under “rural free delivery”. It’s because there was so much “rural” and so little “town”. I bet one extra letter carrier in a truck could have covered the town if they had wanted to.
My parents got so tired of having to go pick up mail during post office hours that they gave me the key as soon as I was tall enough to reach the box, since I passed the post office on my way home from school.
Perhaps if the government is so concerned about PO boxes and Homeland Security, they could give the USPS a few extra bucks to hire a few more letter carriers for these small towns so they could start mail service there.