As someone else said upthread, I don’t think there is a good response. I understand that it is annoying to be asked repeatedly, but it’s also important to realize that each person is, unfortunately, ignorant of how often you get asked those sorts of things. For instance, I don’t have an accent, but my first name, while well known, isn’t terribly common, and matches with a well known TV character that most people my age are familiar with. As such, more often than not, when I meet someone roughly my age, they think they’re clever when they say the catch phrase associated with that character. Yeah, it’s not the same thing, but it also highlights that, from my perspective, it’s annoying because I hear it constantly, but from their perspective, they probably don’t know more than a couple people with that name, so it seems interesting. As such, I’ve tried to learn to be more conscious of those sorts of things and, for instance, will not randomly ask a stranger where they’re from.
That said, the reason that there’s no good answer is precisely because these people have no association. Nothing you say to one will make it any different the next time someone is curious. So, you can either choose to be polite, if very brief, or you can snap back with a clever response and potentially have an unnecessarily negative interaction, and possibly even reinforce some negative stereotypes about wherever you’re from.
Moreso, if you’re actually having some idle chatter with someone, rather than a random stranger butting in, it’s just an unavoidable question. Where such small talk would often include the weather or the price of gas, if there’s something that piques someone interest, it’s likely going to get asked. In your case, it’s your accent, but the same will happen with any number of things, like a particularly unusual hairstyle or hair color, an unusual form of dress, or something that obviously indicates some sort of other topic like association with a sports team. If anything, you should count yourself fortunate to not get stuck to such boring topics as “boy, it sure is hot out today!” and instead will often have the chance in such occassions to either brush it off with a “I’m from wherever” or, if you feel particularly social, actually going on and talking a bit about it.
For instance, I’m obviously in good shape, so I often get asked in idle small talk “How much do you bench?”, and it gets a little annoying sometimes because of how predictable it is, and depending on my mood, I may just give a quick response, or if I can use it as a launch point for a slightly more interesting conversation. It’s a hell of a lot better than the weather conversation, which is ALWAYS boring and never goes in an interesting direction. So, sure, even if it is a bit annoying, consider the alternative.
TLDR: There’s no good answer, so you might as well roll with it.