How does GABA, opposed to serotonin, work to reduce anxiety and/or depression?
That question is a whole lot harder than you think. You would have to understand the entire brain system to know the whole answer and no one has ever come close to that. GABA receptors are the primary inhibitory mechanisms for slowing neuron firing. There are several subtypes of GABA receptors and drugs like the benzodiazepines and alcohol affect them to slow down some neuronal activity. That temporarily alleviates some types of anxiety.
Serotonin and its receptors are a completely different system. No one truly understands how manipulating them affects depression. I don’t want to sound flip but it would take many graduate level courses on this topic to begin to get an understanding of it and, even then, you still wouldn’t have the final answer. Neuroscience is pretty primitive compared to other scientific fields because it deals with the most complex topic in the known universe and it takes a long time to make significant progress. That is a big reason why I dropped out of bahavioral neuroscience Ph.D. program. I wanted to find out the answers to questions like yours and they just aren’t there.
Both GABA and serotonin are neurotransmitters that are created in your brain and play a fundamental role in its workings. Furthermore, it is not simply a matter of having the right amounts of them around, they need to be there in the right (very tiny) amount, at exactly the right places, at just the right times. If your GABA (or serotonin, or other neurotransmitter) metabolism is messed up somehow, your brain will not function as it should, and, no doubt, depression is one possible symptom of that.