“Sorry, no” can be perfectly nice or curt and somewhat rude depending on the situation.
“Do you know what street the closest bus stop is?” “Sorry, no” is perfectly fine.
“Can I borrow your pen?” “Sorry, no” would be somewhat rude.
I guess it becomes borderline rude when the usual answer is “yes”. Most people who have dogs out and about the general public are happy to have them petted by nice people. It’s ok if you don’t want to have your dog petted, but the expected answer is yes. A word of explanation makes it a courteous response.