The problem you are making is assuming that eggs and sperm ever existed separately in a population. That’s not the case. And that also answers your question. You couldn’t ever go back to a time when eggs existed but not sperm.
I think the thing you need to realise is that organisms like humans, where one individual produces sperm and the other produces eggs, are fairly rare. Most species produce both eggs and sperm in the one individual. If you went back in time to the arrival of the first cell that could be called egg or sperm then there would be both eggs and sperm. It would probably help to understand what eggs and sperm are and how they came to be.
In simplified terms the first organisms to undergo sexual reproduction simply stumbled into one another and exchanged genetic material. That’s OK for those organisms that were highly mobile or had huge population densities, but it’s pretty useless for things like sponges or coral or algae that grow on rocks. Instead these organisms started producing cells that could swim around all by themselves. So while an individual coral or alga isn’t mobile it could produce these cells as sort of colonists that could go out to parties and meet interesting people.
Those first mobile reproductive cells were the forerunners of eggs and sperm, and they were indistinguishable. Realise that these first mobile independent cells were released straight into the water. The idea of internal fertilisation is a recent evolutionary novelty and it relies on the same basic methods used for external fertilisation.
Also realise that no matter what their genetic makeup these cells were the same physically. There are still some organisms where the different sex cells are identical. Obviously there’s no point referring to male and female when the cells are identical so they simply get designated + and – strains. + can only fertilise – or vice versa. The problem with that solution is that it’s inefficient. Every cell is able to do two half-arsed jobs. They can all swim in search of mates, but they are far too heavy to swim very fast or very far. They all contain large, heavy fat and starch reserves to allow the zygote to develop to the point where it can feed itself, but they don’t contain very much because if they did they couldn’t move. It’s not really a very good system. It works, but there are obvious flaws with a one-size-fits-both approach.
So inevitably some organisms evolved a novel approach. Instead of having uniform +_ and – cells they developed two distinct cells types. One cell type became ridiculously large and bloated with fat or starch reserves. That allowed any zygotes produced to grow faster and survive far longer without feeding. The downside of course was that they became so heavy that they couldn’t effectively power themselves. They were at the mercy of the ocean currents. The other cell strain took the opposite approach. They became streamlined and hydrodynamic. The cilia or whatever was used for locomotion became much larger and more effective. Internal energy supplies were stripped away in favour of sugar supplies for rapid energy provision. These things became the equivalent of sprint cars, good for fast short runs but with no long term survival potential.
And that was the origin of eggs and sperm. The strain that was sedentary is what we call the eggs or more correctly ova. The motile strain is what we refer to as sperm. It’s really that simple. Although the system has been refined over the eons there is still no fundamental difference between eggs and sperm beyond one being laden with fat/starch supplies and largely immobile and the other being lightweight and highly motile. As with most things in evolutionary history there’s no hard and fast line to divide them. You can define eggs and sperm wherever you like along a spectrum. Some species will have a strain that is 1% lighter and more mobile than the other while other species have a strain that is billions of times lighter and infinitely more mobile.
You can choose to define sperm and ovum wherever you like along that spectrum. But no matter where you draw the line the fact that you have defined one strain as being more motile and thus sperm means that the other strain must be less motile and hence ovum. Basically egg and sperm are privatives. You can’t have one without the other because they are defined with reference to one another. You only know that a strain of sex cell is an ovum because you have defined the other as sperm or vice versa. If you only have one type of sex cell hen it’s neither sperm nor ovum, it’s simply a sex cell.