The earth spins at about 1600 km/hour at the equator. This speed, naturally, decreases with distance from the equator till it reaches zero at either pole. Additionally, sea level is a little closer to the centre of the earth at the poles than it is at the equator.
Which means that, all else being equal, someone travelling towards the equator (either due south or due north - depending on hemisphere), will also necessarily be gaining eastward velocity and going “uphill”. Does that mean that it takes more energy to travel towards the equator than away from it along the surface of the earth?
What would happen if you were on a perfectly spherical Earth spinning at the same rate as our ellipsoidal one? As you walk from the pole toward the equator, you gain eastward velocity just as you said and therefore the Earth has to do (physicist style) work to accelerate you. Walking the other way, you lose velocity and the planet is doing negative work. I didn’t say anything about gravity because, on the spherical Earth, it never makes any difference; the force of gravity is the same wherever you are. When you add the gravitational force and the centrifugal force, (in the opposite direction) the total force is less the farther you are from a pole.
In other words, on spherical Earth, when both gravity and centrifugal force are considered, walking toward the equator is walking “downhill” because the spinning Earth is doing part of the work.
Spill a glass of water somewhere near the pole and watch what happens. The turning of the planet flings the water outward. It starts to run “downhill” until it gets to the equator.
Spill more glasses of water and it starts to pile up along the equator. Because the force of gravity is less farther from the centre, piling up is moving in the direction of least potential energy. Spill more and more and the water finds its level. I.e. it spreads out until there is no tendency for the water to flow in one direction or the other. I.e. the sum of the gravitational force and the centrifugal force is the same at every point on the water’s surface. The ellipsoidal surface of the water looks a lot (well, exactly) like sea level here on real earth. The water at the equator is farther from the centre of the Earth by just enough to counteract the centrifugal force.
Pushing the water down somewhere causes it to be pushed up somewhere else. The pushed up water can lose energy by flowing down until it’s ellipsoidal again and, in absence of any other force, it will do so.
Spherical Earth is, of course, made of unobtainium. Any other substance, the size of and subject to the magnitude of forces we’re talking about will deform until it’s pretty ellipsoidal. If the substance is rigid enough, the surface of the ellipsoid might be bumpy but will follow closely to the ellipsoid. Earth looks pretty bumpy when you’re standing in the middle of the Himalayas but from out in space it’s pretty darn smooth.
So no, when you account for both gravity and spin, you’re not walking up or downhill when you walk along the ellipsoid. The ellipsoid is actually the Earth’s way of making sure that it takes the same amount of energy to go to the equator from the pole as it takes to go back.