The issue is that the 4 GB of address space has to cover everything addressable by the CPU–and that includes things that are not main memory. In particular, in those days video memory would be fully mapped by the CPU. If you had a 512 MB video card, then you would lose 512 MB of address space. More if the video card had more memory. Darren_Garrison possibly had a 1.5 GB video card, or two 768 MB cards, or perhaps a 1 GB card and some other device that used address space.
64-bit systems have plenty of address space to go around. And eventually (I should remember when, but I forget the timeframe), video card drivers stopped mapping all of video memory, and instead just had a relatively small (usually 256 MB) segment of mapped memory. Though this is changing now with “resizable BAR”.