Adding more pigments to the eye does increase the number of colors you can distinguish but at some point you start to get diminishing returns. Three pigments are a big improvement over two, but four provides a much smaller improvement over three. From an evolutionary perspective the small increase in color discrimination provided by having more than three receptors probably isn’t worth the additional wiring required.
The problem with two receptors is that any color can be mimicked by a single monochromatic source of the appropriate intensity. Two receptors are hardly better than one. It takes a minimum of three receptors to reliably distinguish different spectral distributions.
There was a thread around here a while ago about “tetrachromats”, people with four sets of colour receptors. Unfortunately, none of them were ultraviolet.
There’s an optics book called “Seeing the Light. Optics in Nature, Photography, Color, Vision and Holography.” by David Falk, Dieter Brill and David Stork published by Harper and Row in 1986 that is an excellent beginner’s book on optics.