Nearly all the red-sensitive and green-sensitive cone cells are densely packed together in, or close to, the fovea, which corresponds to only about 2degrees of visual angle at the center of your visual field. The whole visual field is about 200 degrees, but in fact there is very little capacity for color vision over most of it. Although the blue-sensitive cones are distributed more widely, the vast majority of the of the receptors in the periphery are rods, which are not involved in color vision.
So, really you can only see things in full color when you look pretty much straight at them. I know it does not seem like that, but color vision in the periphery is largely an illusion. Part of the reason for the illusion is that normally we move our eyes about very easily and freely (and mostly unconsciously) which means that as soon as you think to check whether any part of the scene in front of you is colored, you automatically turn your eyes to look at it, focusing it onto the fovea, and so see its colors. I do not think that is the whole story, though, because I (and, I think, most people) find that if I deliberately hold my eyes as steady as I can, staring at the same spot, I still have the impression that the periphery of my visual field is colored. However, this is an illusion. We may “know” what the colors in the rest of the periphery are because we have previously looked, but we do not truly see them. You can demonstrate this lack of color and detail vision in the periphery to yourself by staring straight ahead, keeping your eyes as still as possible (it is impossible to hold your eyes totally still, but you can get near enough for this), and bringing an object, whose color you do not know (this is important, because otherwise you may “imagine” you can sense its color long before you really can), gradually into range of your peripheral vision, and then slowly further in towards the center of your gaze. You should be able to see that something is there long before you can say either what it is (unless you already know) or what color it is, and even when you can first detect the color, it will probably not seem nearly as vibrant as it does when it is right in front of you, fully foveated.
As far as the foveal area itself goes, in normally sighted people the red-sensitive and green-sensitive cones there are distributed very irregularly and unevenly (regardless of sex) and quite differently in different individuals. Although the blue sensitive cones are spaced fairly regularly, in some regions of the fovea there may be relatively large patches containing red cones but no green ones, and vice versa, or in other parts they may be more intermingled. It seems to be fairly random (but with a tendency for the same type to clump together). Furthermore, some people seem to have a noticeably higher proportion of the red-sensitive or the green sensitive type. But we are talking about people with perfectly normal color vision here. These differences between people’s retinas, and across an individual’s retina, do not seem to have any effect on a person’s ability to discriminate colors. This is probably because the eyes are in constant rapid motion (even when you think you are holding them still) so that any point in the visual field will have been focused on many different cone cells before the brain (to put things a bit crudely) has got around to figuring out the color at that point.
So, all in all, yes you are right that women will have their red-sensitive and green-sensitive cone cells unevenly distributed in their retinas, but men do too, and it does not matter a damn.
On the distribution of cones see:
Conway, B.R. (2009). Color Vision, Cones, and Color-Coding in the Cortex. The Neuroscientist, 15 #3, 247-290.
Roorda, A. & Williams, D.R. (1999). The Arrangement of the Three Cone Classes in the Living Human Eye. Nature, 397 #6719, 520-522.
[Both paywalled I am afraid, but the second has some amazing pictures of people’s retinas, showing the arrangement of the different classes of cone.]
On eye movements in general, and the, basic structure of the retina see:
Richardson, D.C. & Spivey, M.J. (2004). Eye Tracking. In G.E. Wnek and G.L. Bowlin (Eds.), Encyclopedia of Biomaterials and Biomedical Engineering (pp. 568-582). New York: Marcel Dekker. [PDF]
On the importance of the constant small movements the eyes make even when we are trying to hold them still (but not speciffically concerned with color vision) see:
Martinez-Conde, S. & Macknik, S.L. (2007). Windows on the Mind. Scientific American, August 2007, 56-63. [large and colorful PDF]
Martinez-Conde, S., Macknik, S.L., & Hubel, D.H. (2004). The Role of Fixational Eye Movements in Visual Perception. Nature Reviews: Neuroscience, 5, 229-240. [PDF]
Rucci, M. & Desbordes, G. (2003). Contributions of Fixational Eye Movements to the Discrimination of Briefly Presented Stimuli. Journal of Vision, 3 #11, Article 18, 852-864.
For some work dealing more specifically with the role of eye movements in color vision, see:
Bompas, A., & O’Regan, J.K. (2006). More evidence for sensorimotor adaptation in color perception. Journal of Vision, 6 #2, 145-153.