The best defense at the early stage is to have no troops and a big cranny. Potential raiders will be discouraged by coming away empty. Unless they’re like me and they keep forgetting which towns are productive and which aren’t.
The exception of course is if you are near some asshole who just wants to make your life miserable.