Shot in the dark here but the examples with “im/in” seem to have not only come from Latin (well really medieval French) but were actually made in Latin. What I mean is that maybe examples where we use “un” are for words that we didn’t inherit from Latin or did so more recently, whereas examples with “in”, such as “possible” vs. “impossible” were already present in Latin when it bequeathed them to it’s daughter languages.
Mostly though, there’s not much rhyme or reason to it.
(note: I’m taking “im-” and “in-” to be the same underlying suffix which is changed based on the phonological environment in the surface word).
I always took “invaluable” to refer to something that literally cannot be assigned a value. Even, say, a diamond that’s worth 100 billion dollars in not invaluable, because its value is 100 billion dollars.
But a one-of-a-kind diamond so rare and special that nobody can determine its comparable worth truly is invaluable.
Or “Your friendship is invaluable to me” means “I couldn’t possibly put a price on our friendship.”
That’s more accurate than “very valuable,” but even then in- is still being used as an intensifier. The thing being described is so valuable that it can’t be valued easily on the scales we typically deal with. That’s definitely a different meaning from simply “not valuable.”