Given a plane with the three points A B and C, using only formulae,
how can one tell if point P is on the plane, and if not, which side
of the plane it is on.
You want to take the triple scalar product of three vectors, specifically, PA, PB, and PC, or something like that. The formula is:
TSP = PA • (PB × PC)
It can also be written in matrix form. Write the components of the three vectors as a 3 × 3 matrix and take the determinant:
TSP = determinant of…
PA[sub]x[/sub] PA[sub]y[/sub] PA[sub]z[/sub]
PB[sub]x[/sub] PB[sub]y[/sub] PB[sub]z[/sub]
PC[sub]x[/sub] PC[sub]y[/sub] PC[sub]z[/sub]
Now then, is P on top of the plane or on the bottom? Well, what do we mean by “top” and “bottom”? Let’s say you’re looking at the plane from one side, and you connect the dots A, B, C, then back to A. If you connected them clockwise, we’ll call the side that you’re on the “top”, and if you connected them counterclockwise, we’ll call that the “bottom”. Then, based on the sign of the TSP, you can determine where P is with respect to the ABC plane:
TSP > 0 : P is on the top side
TSP = 0 : P is in the plane
TSP < 0 : P is on the bottom side
By the way, thanks for asking a math question. Those three from yesterday really got me in the spirit. 
Well, I don’t have any formulae (and since geometry is based largely on theorems and postulates, I don’t see why you’d necessarily need formulae), but I just tried out a method that I think works; hopefully I can convey it properly.
Take points A, B, and C. Draw lines AB, AC, and BC. Now draw lines perpendicular to AB, AC, and BC that intersect their respective lines at points A, B, and C. You’ll have a total of nine lines (fewer if you have points positioned such that some lines are coincidental). Now, plot point P. If line AP intersects any of the lines we’ve drawn on plane ABC anywhere other than point A, then P must be on the plane. If not, then point P is not on point ABC.
I can’t prove that this works, but it has for the models I’ve drawn. As for which “side” of the plane point P is on, isn’t that a matter of our point of view?
On preview, it seems Achernar has beaten me to it and done a better job, at that. Good stuff, Achernar.
Are you drawing models in three dimensions?? Hey, let me have a turn.
I get a sign reversal with those, using A(0,0,0), B(1,0,0), C(0,1,0), P(0,0,1)
I would have used a different set of vectors. The cross product of AB and AC is a vector perpendicular to the plane of ABC, at A, in the direction of the right hand rule. If you then take the dot product of that vector with the vector AP, it’ll tell you, by the sign, whether AP is in the same direction as the cross product or not. If it’s zero, it’s in the plane.
Sorta an explanation of why the triple product rule works.
I could be wrong, but I get PA = [0, 0, -1], PB = [1, 0, -1], PC = [0, 1, -1], TSP = -1, which is right because P is on the “bottom” side of the ABC plane.
There are indeed a bunch of different ways you could set up the TSP. Starting from any of the four points, draw vectors to the other three, and then use those three in any order. You’ll still get the same magnitude; the only difference will (possibly) be a sign change. I chose to start with P because I liked the symmettry, but if you prefer A (or B or C) that works just as well.
D’oh! :smack:
Well, I drew two dimensions; I had to settle with holding my pen above the plane and eyeballing it for the third, though I guess I could’ve used a straightened-out paper clip if I were feeling frisky.
In any case, does my method actually work?
Yes, your method works, but you could get away with only two lines in the beginning instead of nine, if you could guarantee that the lines were not parallel, and that neither of them passed through point A. If you draw line BC, then the perpendicular lines to BC that pass through B and C, that should suffice. Any line in the plane will have to intersect one of those three lines at a point other than A.
That is indeed the way you defined it. I’m just not used to having i x j = k point down.
I don’t think you get the same magnitude all the time, if you use any of the four points in any order, do you?
If there’s a magnitude change, it’s from the cross product. The dot product can only change signs.
That’s why I didn’t call them “up” and “down”; I called them “top” and “bottom”. 
It would not be difficult to distinguish the “more z-ward” side from the “less z-ward” side, or the same with the x or y axes. One small problem is, though, that whatever method you pick degenerates if the plane is parallel to the axis of your choice. So a vertical plane does not allow a distinction of “up” and “down”. My definition allows a distinction between the two sides, as long as A, B, and C are noncolinear. Of course, talking about the ABC plane when they are collinear is meaningless as well.
Perhaps I wasn’t clear. You must start each of the three vectors at one of the points. So you could use CA, CP, and CB, but not CA, AB, and BP. Geometrically, the TSP is the volume of the paralellipiped with edges defined by the three vectors. You can also think of this as six times the volume of the tetrahedron with those four points. Since the volume of the tetrahedron does not depend on which vertex you start from, it should be the same starting from any.