The insolubility is exactly the problem. The calcium gets bound up in salts, instead of being where it should be - dissolved in your blood and keeping your muscles working, especially the heart.
And HF does burn, perhaps not as much as other acids, but it does cause pain and skin lesions in sufficient concentrations.