I got it working and it sounds 100x better, thanks a lot you guys. 
So what fixed it?
I added the ground from the source (game console) to the grounds I already had on the LM380 out of laziness. I wonder if it could be improved by removing the grounding to the switching power supply, but it’s night and day compared to before where it was completely unplayable.
I will test when I have access to an actual cabinet with a volume pot instead of my current jerry rigged test setup, but it’s already more than decent as-is!
People in the arcade scene are fond of saying “ground is ground” but that definitely wasn’t the case here.
It never is. Indeed there is hardly an aspect of electronics or electrical systems where thinking like that won’t get you into significant trouble.
One critical point is to realise that all voltage is a difference between two nodes, and current is a flow around a circuit. There is no such thing as voltage without two points to measure it between, and no such thing as current that does not complete a full circuit. By convention most circuits call the reference level - the zero volt reference - ground. There is nothing special about this reference, but sometimes people seem to think that ground is somehow magical and pervasive, and that even without a clear path, ground still exists and can be referenced to. But unless you have a solid story about how the reference is maintained, it won’t be so. Same problem then happens with the current flowing though the circuits - somehow this magic pervasive ground will automatically return current around the circuit. Which it won’t. Not unless there is a clear and well defined path.