I tried a little experiment today. I imagined a rock in my hand, then a stone.
In my mind, the rock was rough, while the stone was smooth.
This would work with stones that are smoothed by natural processes as well as human intervention. Exposed sandstone and limestone are going to be smooth.
Gemstones may be faceted nowadays, but they used to be polished.