I don’t understand how it could possibly mean what you guys say it “really” means. At some point, someone must’ve gotten it backwards and that became popular. I didn’t have to “try” to interpret it logically, it’s just what the words plainly mean.
If I say I tell someone that I came to my hometown from Springfield by way of Branson, I don’t mean I was first in Branson. Why this would change if Springfield is my birth town, I just don’t get.
I thought it was “plain” at first too (and I did wonder, as you hint, whether it had an origin like “I could care less”); but mulling it more I don’t think it’s quite that clear. Think about the possibility of “by way of” referring to the place that delivered you (literally, from your mother) to the place you were later.
As you see, I can argue it either way at this point!
If a guy says “My family came to America from Europe, by boat,” you’d understand him just fine. If a guy says “My family came to America from boat, by Europe,” you’d back away slowly and look for a weapon, because he’s spouting deranged nonsense.
Ah, but there’s that tricky “way of” business which muddies the waters.
As an aside: judging by what there is out there on Google, on word reference forums and so on, we are doing a service in terms of “fighting ignorance”. Unless it was several pages into the search results (and really: who looks past the first page or two of Google search results?), there is great linguistic misunderstanding about this phrase out there. Hopefully this thread will become a significant hit for the search, because otherwise I don’t think foreign language learners or linguistic historians of the future will get much of a sense of how this phrase is used on the ground versus how much confusion it engenders.
Granted – but while it’s a small but pretentious step to “I came to America from Europe by way of the Concorde”, no one would express that sentiment as “I came to America from the Concorde, by way of Europe.”
The first is muddier. The second is mud.
I’d actually interpreted a completely different way
X by way of Y
I got to Z directly from X, passing through Y first, but I initially came from some unspecified initial point A.
That is, there is no starting point, the starting point is unspecified.
“By way of” is a routing point, but to me the phrase fairly clearly and unambiguously chains [somehwere]->Y->X->[here]. The more natural, complete phrase to me would be “I was born in Indonesia, and I came here from California, by way of Japan.”
Incidentally, imagine we meet in England and you casually ask how I came to be there; “I swam from France” seems like an unobjectionable reply.
“I swam from the English Channel” sounds inaccurate and stupid; I didn’t swim from the English Channel, I swam while I was in it and stopped when I was done.
And so “I swam from the English Channel, by way of France” is presumably wrong, because that first part is wrong. But start with “I swam from France,” and specify that you did so “by way of the English Channel” – well, that sounds fine and might be correct, while the alternative doesn’t and isn’t. Lesser of two evils, y’know?
But we’re talking about “from”, not “came to”. Though that does raise an interesting point: technically “by way of” is a adverbial prepositional phrase that one normally expects to modify a verb that describes travel, or motion, or transferral, or something along those lines. But in the “I’m from X by way of Y”, we don’t have that–and this creates some of the ambiguity.
One thing that occurred to me today was that if I’m right in my supposition that this phrase is used more by rural and Southern folks, it may be more common among the religious. And so they might (I’m a lifelong atheist, so forgive me if my speculation here is off target) have a sense that someone’s soul actually starts out somewhere else, like in heaven or something, before they are born. Then it might make more sense that the place they were born is a waypoint they would refer to as “by way of”. They started in heaven, were born in Seattle (oops, picked the most secular large city in the U.S. without thinking there), and now live in Chicago but are speaking to people in Albuquerque. So they are “from Chicago” (as in, that is where I came to you from and where I will go back to tomorrow), but got there “by way of Seattle” (the place they arrived in this world from the place where souls start off).
One other random observation: the proportion in the poll has stayed remarkably close to 80/20 ever since the number of total votes were maybe around 30 or so. Right now it is exactly 100 votes, and: 80/20. Neat. 
From my perspective, it dispels the ambiguity.
Start at X and go to Z? I’d say I came from X.
Start at X and go to Z via method Y? I’d say I came from X by way of Y.
Start at X and go to Z by passing through Y? I’d say I came from X by way of Y.
This sounded so convincing, I momentarily wondered how I had ever been able to see the other side. But then I realised you are adding a verb of motion: “came”. Not to mention the prepositional phrase “to Z”. But the idiom is not “I came from X to Z by way of Y” but rather (from someone speaking in the setting of Z) “I am from X by way of Y”. And that, again, is where the ambiguity creeps in.