I know how in principle, what I mean is how the hell did it not take weeks or months of non-stop grinding and polishing? The few times I’ve tried I gave up because it would apparently take hundreds if not thousands of man-hours. I know there are telescope buffs who hand-grind and polish lenses and the heck if I know how they do it.
… elbow grease …
It did take weeks and months of non-stop grinding. But what was the alternative? You did it or had nothing.
Things happened at a different pace.
I remember reading a passage where Ernest Rutherford paid a visit to Charles Wilson, and talked to him while Wilson sat on stool, meticulously polishing the cylinder for his cloud chamber. He came back some six months later, to find Wilson in the same spot, still polishing the same cylinder…
Not quite polishing but I once watched a neighbor take a solid steel bar and turn it into an octagonal flintlock barrel with nothing but a file and hand operated drill.
Took him all of two days.
“By hand” doesn’t necessarily mean just with a file in your hand.
I’ve sharpened knives with an antique sharpening wheel – a big stone about 3 feet in diameter, moved by bicycle-like pedals attached to it. Like this. It’s quite heavy, but once you got it moving, it didn’t take all that much effort to keep it going at speed as you sharpened on it. I imagine there were similar horizontal ones for polishing stones or gems.
What else did people have to do back then? Hell, this was their entertainment.
A flat stone surface takes only a day, if it wasn’t too hard and you have the right abrasives. For metal, I suppose a Japanese sword polisher is the best example. Two weeks if it’s a newly forged and hardened katana. A few days if it’s a badly rusted and scratched sword.
Not 5000 years ago, when people were making polished stone axeheads.
They used spit and cloth and ashes. They knew the use of ashes.
Lapping, where two surfaces are rubbed together with an abrasive between them.
While the polissoir is the name for an early woodworking tool the method is similar and it will provide a good google image search.
Even today using tin or MDF to polish hard metals like tool steel is one of the most effective methods.
I haven’t polished rock but I have polished metal when making jewelry. It is tedious but a lot of the steps in handcrafting jewelry are equally tedious. It’s one of those things you just do it till it’s done. In jewelry it actually gets harder as you go along. There are some chemicals that help, but I forget what they are.
I have one of those. Just the wheel and axle, no frame. Belonged to my grandfather, who knows how long it was in his family. I’ve always wanted to reconstruct the rest of it so that it would be functional again.
Back in the seventies if you wanted to grind out rock or metal you had to do it without tools like Pro Tools.
As a project for first year astronomy, I ordered a kit (from Edmund Scientific) and ground out a 4.5 inch mirror for a telescope. It took about a week of steady work, a few hours at a time, using progressively smaller grit.
In some of my reading, one of the questions was - how do you make separate grades of grit from scratch? The answer was to mix the powder in water and different sizes took different times to settle onto the bottom. Or, you order a kit where they are already separately graded.
A more interesting consideration is how did they precisely cut tens of thousands of giant stones from bedrock to make the pyramids, in the millennia before iron tools? Most likely, pointy rocks, copper chisels, wetting wood wedges - and a lot of hammering.
IIRC all those renaissance shiny marble statues were polished with hand tools, then sanding, then polishing - one trick was finishing with leather and embedded iron rouge grit.
Bonham has the best point - even in the good old days, spinning wheels provided the equivalent of power sanding for small items when necessary.
Slightly related … if you want to be a woodworker … first learn to build a house, find a wife, have babies … and teach your children how to sand …
In high school I ground and polished an 8 inch telescope mirror. I worked on it for an hour or two most days. Unfortunately the pyrex blank I started with was slightly convex, so it took a two or three months to bring it to the right shape with the coarsest grit. Then more months of going through progressively finer grits, then polishing with rouge on a pitch lap, and finally the final figuring to convert it from a spherical shape to a paraboloid. The whole process took almost a year. So the answer to the OP’s question “how did they do it” is “persistence”.
In the good old days - persistence and assistance. Apprentices will work for food and lodging.
But you did have tools like these. They helped. 
