It’s just slang, not really vulgar.
“Foutre” and “ficher” both originally meant “to fuck”. In things like 18th century pornographic litterature. But neither is currently used with this meaning. I’d guess that a majority wouldn’t known about “foutre”, and almost nobody would know about “ficher”, much rarer even in the past.
I guess that, maybe, in the 50s, “je m’en fous” would have been considered vulgar in good society. And “je m’en fiche” would have been more acceptable. Nowadays, it’s just low-key. Not elegant.
To express the same thing more elegantly you can say “je m’en moque”. Or if you want, at the contrary, to be vulgar, you can say “je m’en branle”.