It’s Memorial Day weekend that the Outer Banks starts getting heinously busy. Before that and after Labor Day are pretty nice.
Nag’s Head, Duck, etc. are, as KneadToKnow said, quite overdeveloped, which is really too bad, IMO, but on the other hand, it might be fine for you depending on what you want to do. As has already been said, it’ll be too cold for swimming, so I assume you’ve got other activities in mind.
A lot of our seafood here comes from the Core Sound, with the fishermen living on Harker’s Island, so you will find some good eats out that way. Clams, bluefish, and sea bass should be in season then.