Then you have Mr Student and his t-test.
(Which does have a rather satisfying etymology.)
Wikipedia has this to say:
Since its introduction, the normal distribution has been known by many different names: the law of error, the law of facility of errors, Laplace’s second law, Gaussian law, etc. Gauss himself apparently coined the term with reference to the “normal equations” involved in its applications, with normal having its technical meaning of orthogonal rather than “usual”.[76] However, by the end of the 19th century some authors[note 5] had started using the name normal distribution , where the word “normal” was used as an adjective – the term now being seen as a reflection of the fact that this distribution was seen as typical, common – and thus “normal”. Peirce (one of those authors) once defined “normal” thus: “…the ‘normal’ is not the average (or any other kind of mean) of what actually occurs, but of what would , in the long run, occur under certain circumstances.”[77] Around the turn of the 20th century Pearson popularized the term normal as a designation for this distribution.[78]
The first citation ([76]) says this:
Gauss (1823) called the system of equations […], which give the least squares parameter estimates […], the “normal equations” and the ellipsoid of constant probability density was called the “normal surface.” It appears that somehow the name got transferred from the equations to the sampling distribution that leads to those equations. Presumably, Gauss meant “normal” in its mathematical sense of “perpendicular” expressing the geometric meaning of those equations.
I don’t see how it’s correct to ascribe any specific meaning to the “normal” in the normal distribution. Gauss may have meant it that way when referring to some specific equations, but he wasn’t naming the normal distribution. The name may have been transferred in that fashion, but later usage didn’t consider that, and it seems likely that popularization of the name may have been more from the “usual, ordinary” sense of the word.
Given the convoluted history, it’s probably for the best to take the name as no more than a name.
The Gaussian distribution does have some “perpendicular” qualities. For instance, if you draw x and y each from the same Gaussian, then the two-dimensional distribution of ordered pairs so obtained will be circularly symmetric: That won’t happen with most distributions.
It does, but as OldGuy noted, that’s not exclusive to the normal distribution. We should be calling it a normal distribution in that case. Orthogonality of some key properties doesn’t narrow things down enough to get you the Gaussian specifically.
One way to arrive at the Gaussian distribution is to look at Euclidean space, let’s say 1-dimensional, and consider the heat kernel corresponding to the heat equation
\frac{\partial}{\partial t}u(x,t) = \Delta u(x,t)
where \Delta=\frac{\partial^2}{\partial x^2} is the Laplacian operator. If we start with a delta function distribution at t=0, the solution is
\frac{1}{\sqrt{4\pi t}}e^{-x^2/4t}.
This works in any number of dimensions—the heat just spreads out evenly. I am not sure what is the precise history of this problem or whether it has to do with (at least Gauss’s) idea of “normality”, though.
Has the name “Gaussian” ever suggested a distribution different from the one we’re discussing most here?
At work I tend to use the Student’s t-distribution when I collect data, since our sample sizes are usually small. Plus the history of the Student’s t-distribution is cool 
.