My two cents: As has been pointed out, the rust on the surface inhibits the further corrosion of the rail. So why not paint the rail at the factory and have no rust at all? It turns out there is a good reason to leave steel uncoated. Along with rail, couplers, wheels, and axles are left unpainted (actually painting is prohibited by the FAR) so that any structural defects that develop over time can be detected. The steel alloys used have a nice tight oxide that acts as a barrier, as has been mentioned.
Absimia mentioned the chunks left over from a rail grinding operation. I believe any sizable pieces were probably from the grinder taking off the burr that forms at the top inside corner of the rail’s cross-section. I have a piece of rail from the old Union Pacific branch line to Ketchum, Idaho that shows a burr about one eighth inch sticking out towards the inside of the rail. Steel can be pretty mushy under big loads.
And speaking of mushy, DougC mentioned the flattening of the rail under load. The tire and rail do flatten against each other under the load of a train. A fully loaded freight car puts around 10 tons on each wheel and produces a flattened area approximately 1/4 inch by 3 inches. That explains why most of the top surface of the rail is shiny, in spite of the fact that it is curved.