Was there ever anything special about the 5th race?

My entire life I’ve heard the saying “Who looks good in the fifth?” I’ve heard this on television shows, movies, even in comic strips. What they are talking about is the 5th race of the day at the race track. Now, I’ve been to quite a few different tracks, and I’ve never noticed anything special or different about the 5th race. Even special races like the Kentucky Derby tend not to be the 5th race.

So how did the 5th race get to be mentioned so much? Was there ever anything special about it?