First, you’ve got to figure out how strong a gravitational field the Earth generates. This is the easy part - just calculate the distance to the Moon, and also its orbital period. Period is one month. Distance is a bit tricky, but you can probably do it with a sort of triangulation. These days you’d use radar or laser. The orbital period depends only on gravitational field and radius of the orbit, so you can calculate the gravitational field.
Note that you do NOT need to know the mass of the Moon for this. It should be easy to imagine that if you split the moon in half and left them next to each other, they would not suddenly start orbiting around the Earth any faster or slower.
OK, so now you’ve measured the gravitational field of the Earth. The next question is, how much mass does it take to generate this field? The conversion factor you need here is the gravitational constant.
This number is rather tricky to measure. You’ve got to take two objects of known masses and measure the gravitational force between them.
(or possibly one object of known mass, and a second object that is much ligher than the first) You can’t use Earth as one of the objects, because you don’t know the mass of the Earth yet. You have to take two big weights into your lab and do your best to measure the almost infinitesimal force between them.