Remember, in the Good Old Days there was not much else to keep people amused; plus, in these days of books, calendars, and smart phone reminders, we forget how much work and detail people could track in their heads. By the time people were settled in cities, they had figured out calendars and how many days per year. (Presumably they had noticed the need for leap year adjustments, too)
As others point out, the behavior of the sun and stars can be easy to track, too, by lining things up and measuring shadows or line-of-sight. I assume the local shaman or priest knew - “the sun rises this far south but no further” since tracking seasons was important to any agricultural society. Once you know northern-most and southern-most travel of the sun, it’s pretty easy to find the middle point. Since the sun’s position is changing pretty quickly then, that’s even easier to pinpoint to the exact day.
So I guess the short answer is - they figured it out.