The Missouri River is the border between Nebraska and Iowa, except for Iowa’s pecker, Carter Lake, which is almost completely surrounded by Omaha, on the Nebraska side of the river. Long ago the river changed course, stranding Carter Lake, which is the name of the town and also a horseshoe shaped lake which is part of the old river.
That was what I was gonna post! Now, all I got is
You can’t get there from here, boy!
Panama is nearly due south of New York.
And of course most of the ones where it’s exactly on the border are in generally flat states where the general trend is just going up. Even if the neighboring state conquered the current high point, the new high point would still be on the border.
Although one interesting exception to all of the above is Boundary Peak, which is the high point in Nevada. It’s pretty much smack dab on the California border, which would make you assume the mountain range simply defines the border, but it’s purely a coincidence. Also, the next highest peak (by less than 100 ft) is Wheeler Peak which is only a few miles from the Utah border way on the other side of the state!
Do you mind my asking, where? According to my atlas, the border between the states runs straight north - south all the way.
At the far northern point, near the NSW triple point. The boarder between Victoria and SA is about 2 miles further west then it should be (141 E). See here.
They say that the border between Idaho & Montana was supposed to follow the Continental Divide all the way up. But the Rocky Mountains branched along the way, and the surveyors took the wrong branch. Eventually they came upon a stream flowing the wrong way, said “Oh, fuck it,” and just headed straight north to the Canadian border.
^ Thanks,** Diceman**, you made my day; I love a good “Oh, fuck it” story, and that one is priceless. 
Except, of course, that it isn’t even close to being true. The boundary between the future states of Idaho and Montana was set at the Bitterroots by Congress in 1864, decades before any surveyor visited the area. When the surveyors did finally visit, in 1904-06, they did their job exactly as prescribed and this is the thanks they get–people accuse them of being incompetent drunkards!
^ Aw, Dad, we was just playin. 
In a sane world, the border between New Jersey and neighboring Delaware would go right down the middle of the Delaware River separating them. But we live in our own Bizarro World, where Delaware actually owns chunks of what you’d think would be New Jersey, on the East side of the Delaware River:
Some of those straight line borders for Arizona, Wyoming, etc are not really defined by meridians and parallels but are defined by surveyors trying to follow the meridian/parallel. The official border is therefore the survey markers and they sometimes missed the intended line by quite a bit, probably depending on the availability of whiskey at the time.
My favorite oddity, which I’ve mentioned before, is the Kentucky Bend. As a result of the meandering Mississippi, the extreme western chunk of Kentucky got isolated from the rest of the state when the Mississippi shifted south, allowing Missouri to spill over into Tennessee and cut off that bit of Kentucky. Apparently the state boundaries were defined by the river, not by what it had been in the past.
I first learned about this from a Ripley’s Believe It Or Not entry, which said that it was impossible to go from the East end of Kentucky to the West end with out crossing another state.
But I have riddled people about this in a more provocative way, asking which were the only three states that met at three distinct and separate places. One guy tried to show me that was topologically impossible, until I showed this to him on a map.
Check out 12 Mile Circle, a blog about all of this stuff.
This was all being conducted with 19th Century surveying equipment. They did the best they could. I believe that US law has always held that the surveyed border trumps whatever the theoretical border should be. That’s why Georgia was SOL in its recent attempt to push its border north and steal water from Tennessee. Regardless of whether or not the border is supposed to be further north, what’s important is that the border isn’t further north, and never has been.
Thanks – fascinating stuff. I now want to visit the extreme north-western corner of Victoria (detail on the ground, basically in too small an area to show up on my Philip’s Great World Atlas) to experience these doings for myself !
There has featured previously on SDMB, the wondrously crazy enclaves-and-exclaves situation around Baarle, up against the Netherlands / Belgian border – which I have been to and sampled for myself.
Re that general part of the world: I was surprised and intrigued to discover a couple of years ago (by chance, in course of a project undertaken) that the New York borough of Staten Island is not only separated by a wide stretch of water from the rest of New York; but is to its west, hard-up-against New Jersey – separated from NJ by only a narrow channel. I’d hitherto imagined Staten Island as being “in splendid isolation”. In practical terms, it might seem to make more sense for S.I. to be part of New Jersey; but how boring and fun-devoid, that would be…
A. W. Von Schmidt’s only excuse was he was just an asshole! 
For the USA, there is a book that details all these stories about states’ borders. I found it fascinating.
How the States Got Their Shapes, by Mark Stein.
The Magdalen Islands of Canada are legally part of Quebec (and, as you might guess, largely French speaking), but the only regular ferry service is from Prince Edward Island. So to get from Montreal to the Magdalen Islands without flying or chartering your own sea voyage, you have to drive Quebec->New Brunswick->PEI->Quebec (Magdalen Islands).