Disclaimer: I’m not a mathematician. I’m sure that any decent topologist or geometer could give you the right answer in a heartbeat.
Is this 4 or 5-D space you speak of Euclidean? If so, there’s a relatively straightforward generalization of the dot (inner) and cross (outer) products: refer to a decent textbook on tensors, like Spiegel’s Vector Analysis. If the space is not Euclidean, then the inner and outer products are properties of the particular space and must be defined.
Realize that there isn’t a 4 or 5-D analog of volume, inasmuch as you can’t visualize it. So calling
*(a . b) x c *
“volume” in a higher-dimensional space is really just wordplay.
You could create a coordinate system where two of the unit vectors are perpendicular to all three vectors, so they can be expressed in terms of the other three unit vectors. Then your (A x B) dot C formula should work, where A, B, and C are in terms of the three vectors that aren’t perpendicular to them. Since you’re only getting rid of one or two dimensions, the Gram-Schmidt process should work well to get you three unit vectors that span A, B, and C.
For the record, this formula isn’t correct; it doesn’t even work in three-space. Consider the vectors (1,0,0), (1,1,0), and (1,1,1). They span a parallelepiped with volume 1, but the formula above gives an answer of sqrt(2/3).
ZenBeam’s method will work fine to get you the volume. Here’s another method: write B=B’+X, where X is the projection of B onto A. Specifically, X=A(A dot B)/(A dot A). Then write C=C’+Y+Z, where Y is the projection of C onto A and Z is the projection of C onto B’. Then the parallelepiped spanned by A, B’, and C’ has the same volume as the original solid, but A, B’, and C’ are all perpendicular to one another, so the volume will be |A||B’||C’|.
(Actually now that I think about it I think this answer is pretty similar to ZenBeam’s…I’ve just written out the first few steps of the Gram-Schmidt process. Oh well.)