You probably know some of this already, but let’s start with the basics. The receptors in your eyes are called rods and cones. The cones are the ones that detect colors and most people have three types of cones. We generally call these cones the red, blue, and green cones, but that’s not quite accurate. The red cones for example will respond to red, yellow, green, and all the way down into the blue part of the spectrum. It’s just that their peak response is in the red section of the spectrum. And the green cones respond to blue, green, yellow, and red light (but again, their peak is in the green part of the spectrum)
Take a look at the image on this page.
As this image shows, the red and the green cones overlap more in their coverage of the spectrum than the green and blue cones do. You will also note that yellow is right in the middle of this overlap between the red and green cones. Yellow light triggers the red and green cones equally, and the only difference with white light is that white light is going to trigger the blue cones more than yellow light will.
So that’s basically why yellow seems closer to white than other colors. You are triggering two out of the three cones in your eyes.
Certain types of color blindness can make yellow and white even less distinguishable. For example, if you have a deficiency or absence of blue cones, then since only your red and green cones work normally, yellow and white will both trigger these red and green cones equally and you will have great difficulty distinguishing between yellow and white.
Some shades of yellow look pale to me, but others don’t. School bus yellow for example doesn’t look at all close to white to me. YMMV (or rather your eyesight may vary).