I appreciate everyone taking the time to respond. It has really helped. I understand the errors in my thinking, but I knew I got the answer right so it didn’t totally feel right to dismiss this method out-of-hand.
The professors solution was a fairly complicated (or rather tedious) expansion of the curl(curl(F)) with the various components of F mixed with all the respective partial derivatives. I could have gotten it right this way, too but as I was working the problem I tried to think outside the box and look at a more clever way to do it. I should have went the “brute force” way because I would have made in A in the course using brute force but instead only got 50% for this problem so I got a B in the class!
For what it’s worth, it occurred to me that if we don’t intend to multiply scalar fields by other scalar fields (as the OP never has to), then we can take scalar field * differential operator to mean the same thing as differential operator * scalar field (that is, the application of the differential operator to the scalar field) with no difficulties.
This corresponds to using a slightly different category than described before: two objects S and D, such that Hom(S, S) = Hom(D, D) = differential operators, while Hom(S, D) = scalar fields and Hom(D, S) is taken to be trivial. This lets us multiply scalar fields by differential operators on either side with the same effect; indeed, this lets us completely rearrange the order of any valid multiplication to get another valid multiplication with the same result (thus, obviating the need to worry about order of factors entirely).
As for the “back cab” formula being valid in the generality of ringoids, presumably, the original proof the OP was given in class of that formula would go through just as well here. For example, perhaps it was done by expanding out the components of A x (B x C) in terms of the components of A, B, and C, which would work just as well in exactly the same way. There are more conceptual approaches to this as well, but they’re no problem either; every approach I can think of transfers straightforwardly to our ringoid context. You just write it down and see that what you’ve written makes sense here just as much as it made sense there.
Indeed, it is not. But as I said, you needn’t understand category theory to understand that the formal manipulations yielding A x (B x C) = … originally would still work just as well in exactly the same way were some of the multiplications interpreted as applications of differential operators.
The category theory is just a way to formalize this after the fact, which is to say, a way to set down new boundaries for this kind of algebra after we’ve realized the old ones were excessively tight. But we don’t actually need to formalize the new boundaries before exploring them. Really, there’s no need to do anything other than say “That old proof of A x (B x C) = …? Substitute ∇ for A, ∇ for B, and F for C throughout that proof, and observe that it still goes through, giving us the desired result”. [The “duck typing” approach to mathematics…]
And if you want some real bastard notations that somehow miraculously work, you need to look to the physicists. We’re always coming up with things like that, and leaving the mathematicians to tsk-tsk us until they can figure out a rationale for them. You do not want to hear the origins of bra-ket notation, for example.
Ah, OK, I was picturing a duck pecking at a keyboard with its bill.
I don’t remember it in all of its gory detail, but the short form of it is, in the expression
<psi|
the < is originally the left half of a <> pair indicating averaging, and the | is originally the left half of an absolute value sign. With a corresponding meaning for |phi>, of course.
