My parents live in Fernandina Beach, FL. It’s on Amelia Island, which is a very nice place - there’s a Ritz Carlton, but it’s not super-touristy. The beaches are very nice but not at all crowded, and the food is amazingly good. There’s some very nice state parks and such, the island has a lot of history if you’re interested, and there’s plenty of stuff nearby but it’s nice and quiet. I love the place.
Try Sanibel Island, down here in South West Florida. Although we’re a booming area, it’s still got a small town feel. There’s Bowman’s Beach on Sanibel and the beaches along Estero Island, Lovers Key, and in Bonita Springs. Lovers Key is the best. It’s very isolated but still close to the towns of Bonita Springs and Fort Myers Beach. The first week of August is during our slow time, so prices are down some. Also, we have SW Florida regional airport (International) in our county so you can get outstanding deals on Airtran in and out of here.
We hope to move to Amelia Island this spring. Its beaches are not quite as nice as where we are now, Port St Joe on the northwest side, but there are a bit more amenities without too much hustle and bustle.
In terms of the Virgin Islands, I preferred the British Virgin Islands, but St John is very nice indeed.
And you can visit Pippi Longstocking.
Oh, wait. That’s Villa Villekulla.
They also have the house they filmed Pippi Longstocking in, weirdly enough, on Amelia Island. It’s nice in that it’s close enough to Jacksonville to go there without living in any sort of a city. And yeah, I grew up going to my grandparents’ in Clearwater, but the Amelia beaches are pretty decent. (Beat the hell out of Myrtle Beach, anyway.) All my favorite restaurants seem to be getting sold, but you can definately eat a lot of really good food still. Unfortunately there’s more and more chains encroaching, but there’s still a definite small town feel to it.
Of course, the real Villa Villekulla is in Kneippbyn. 
You should try Pawley’s Island, SC. It’s pretty quiet and since it’s small, the rentals go fast but the beaches are awesome.
Another great area is Seaside/Grayton Beach in the panhandle of FL.
Also, I’d stay away from Gulf Shores, AL. Around here, it’s known as the Redneck Riveria. The beaches are nice but a bit crowded in the summer.