Why do we use a double quote (") for inches, and a single quote (’) for feet?
We also use (’) for minutes (both temporal and geographical) and (") for seconds. It appears that (’) refers to the first common division–of a yard, a degree, or an hour–and (") refers to the second common division of same.
Remember that “minute” with a long I is a synonym for “very small.”
The measurement of divisions of an hour (hora) was into 60 (primae) minutae partes, which were then in turn divided into 60 secundae minutae partes. Portions of degrees (and IIRC hours at one time) were marked with the single and double ticks now represented by ’ and " in consequence.
I suspect strongly but cannot prove by any cite that the feet-and-inches symbols derive from the minutes-and-seconds one, much as GaryT said.
And a long oo, for those learning English pronunciation at home. 
Actually, it’s the other way around.
The original and proper sign for feet or minutes (or for a first differential) is a little vertical dash, and the single quotation mark is a dot with a tail on it, like a comma. The sign for inches or seconds (or a second differential) is two little vertical dashes, and the double quotation mark is two little marks like commas. What’s more, in an open-quotation mark the tails point up from the dot (and curl clockwise), while in a close-quotation mark they point down from the dot (and curl clockwise). The inverted comma used to mark elllipsis &c. is similar to the closing single quotation mark.
When the typewriter keyboard layout was getting standardised the designers wanted to avoid having to put in separate keys for the open quote and close quote, so they just included a symmetrical dash to be used for both open and close quotes. And it was considered a convenience that this is also the mark for feet (or minutes) and inches (or seconds). This standard was translated into ASCII, and is common (but not universal) in computer typefaces.
So in fact we are using the inch sign for a quote, not the quote sign for an inch.
Regards,
Agback