Suppose I have two colors defined in a Windows application using the RGB form. I want to show an area where they overlap as being the color that would result if I mixed light sources of these two colors. What arithmetic do I use to mix two colors?
Let’s say one color is 255/0/0, or saturated red. The other is 0/0/255, or saturated blue. If I mix red and blue, do I add or average the RGB values? If I add them, I get a 255/0/255, saturated purple with high luminosity. If I average them, I get 128/0/128, a saturated purple with the luminosity cut in half. What would be the actual result of mixing two light sources of these colors?
If I need to add, then how do I mix two colors like 255/0/0 and 255/100/0? I would assume that I would have to add the numbers, then scale them back to get the biggest component back down to 255, to get 255/50/0. If I average I would get 255/50/0 as well.
Since you are pretending that the colors are coming from an additive process (like projection of light from two incoherent sources) the intensities ought to add. Unfortunately, your monitor has a maximum intensity that it can display, so you can’t display the sum of two (255,0,0) reds. Averaging them (as you suggest) is a reasonable compromise.
Note that if instead you are pretending that the colors are being generated by a subtractive process (like colored gelatin filters, or pigments) in which different wavelengths of light are absorbed with different efficiencies, then you should probably multiply the values (scaling 0-255 to 0-1, to represent the absorptivity of each filter).
Complicating all of this is that the color response of monitors is famously nonlinear. If you realy want it to look right on the screen, you have to do more work (and it probably won’t look right when you get a new monitor); for more information see this FAQ on gamma correcction.