Why "of to the salt mines?"

I have heard this expression many times.
What brought this about as another way of saying “I’m off to work”

I can make an educated guess, heck maybe two if I had thought to grab some coffee this morning.

What is the straight dope?
My WAG is salt mines were/are a horrbile nasty place to work.
Saying “Off to the salt mines” implys your going to work as a place you really do not like.
thanks in advance everyone.

Osip

You answered your own question. Check out this
Los Angeles Times article

“Although salt throughout history has been a source of wealth, it’s no accident that we still use the phrase “back to the salt mines” to summon up an image of extremely hard, thankless physical labor. And indeed, as Kurlansky notes, the lot of those who labored in the salt mines was seldom a happy one.”

Brian

According to tradition, the third Pope, Clement I, was banished to the salt mines in Crimea by the Romans.

There’s a great (relatively), low-brow cinematic use of this phrase: in the second James Bond movie, “From Russia with Love,” the younger wife of a middle-aged Turkish man is whining and acting seductive and entreating him to come ‘take care of her needs’. Moments after he leaves his desk to join her in bed, removing his tie and saying to himself “back to the salt mines,” his desk and chair blow up.

The hedonistic message here is that maybe there’s never too much of a good thing. After all, Bond never has to work in the salt mines, though he does have to put his tool to frequent use…

“allez Kerim-Bey!..”

“He said thank you for saving his life, you are now his son.”