Demanding Mistress?

What is the derivation of this phrase? I had an argument a long time ago about whether it was a boss/school mistress or a lover mistress. I can’t seem to get much on google. I’m pretty sure I was right that it means a boss or teacher.

I never considered it to be an actual woman, but instead a hobby or interest or perhaps even a job, that commands the interest of a person yet is often difficult or ‘demanding’ in the sense of challenging or onerous.

So it is neither a boss/teacher nor a lover. Its a metaphor.

A metaphor maybe, but it’s easy to picture some guy whose mistress, jealous of his wife, wants more attention and presents to compensate.

This is the interpretation I have always had.

I think you missed my point? We know it’s a metaphor. The question is what for? The word mistress is in the phrase. I don’t know why. Who is challenging or onerous?

The figurative sense of “mistress” is common enough that it’s even in some dictionaries, eg. Merriam-Webster

ETA: I don’t think the figurative sense derives from mistress in the sense of an extra-marital sexual partner, rather mistress in the sense of a woman with power or authority.

I always heard “harsh mistress.” E.g., the Moon.

MY favorite “Tick” quote “Gravity is a harsh mistress!”

I know I have seen either one of our resident physicists, aviators, or rocket scientists post the more direct “gravity is a bitch.”