The only math which you absolutely need to know for quantum is differential equations and matrix algebra (along with all prerequisites of those, of course: Multi-variable calculus, trig, algebra, etc.). For General Relativity, you’ll need to gain a very strong working knowledge of tensor analysis and differential geometry, but here I wouldn’t recommend taking those math classes. A good GR course or book will, of necessity, start by introducing you to all the math you need to know, and what exactly you need to know about it. A math course on the same subject, however, will spend a lot of time on concepts you’ll never have use for in physics, without covering the practical shortcuts which physicists use. If you have the soul of a mathematician, then you will walk away from a Differential Geometry course with a very deep and satisfying grasp of the material, but if you’re a physicist at heart, it’ll mostly just confuse or bore you. Quick litmus test: When you look at an expression like (dy/dx), do you see “A little piece of y, divided by a little piece of x”? If so, don’t bother with the diiferential geometry courses. A mathemetician would shudder at the thought of treating (dy/dx) that way. Like in quantum mechanics, matrix algebra and differential equations will be useful, but for rather different reasons. For physics prerequisites, a good E&M (Electricity and magnetism) course will also help prepare you for relativity.
There are also a lot of math subjects/classes which won’t be directly useful, but which will indirectly make the physics much easier. I highly recommend that you study abstract algebra/group theory at an undergrad level, for either quantum or GR, and some knowledge of probability/statistics will also help in quantum.
As for books, Liboff and Griffiths both have good undergraduate quantum books, and I would recommend either of them. For GR, MTW (Misner, Thorne, and Wheeler, mentioned above by CalMeacham, is sort of The Bible, and contains everything a person might possibly ever want to know about relativity, but it’s a pretty poor intro. I found that Schutz’s green book was much more accessible (and more portable! You’d get a hernia lugging MTW around), and it uses all of the same notational conventions as MTW, so it’ll make it easier to move up to that tome, should you so decide.
A final recommendation, by the way: It’s much more difficult to learn from a book (or books) than it is from a class. The best way to learn this material would be to go for a graduate physics degree, but short of that, most colleges will also let you sign up for (or at least sit in on) classes without enrolling for a degree. This’ll give you access to a professor to handle specific questions, and will also provide some guidance in assigning (and checking) homework and tests. Yes, classes cost money, but if you try to learn exclusively from the books, you’ll already be shelling out hundreds of dollars: College textbooks typically cost around $80 each.