I recently bought a flat panel display to go with my new computer, and I just realized it only has analog (VGA) input 
My video card does support digital output, so I think my question is obvious…how does performance compare between analog and digital connections? Is it worth going back and exchanging the display for another (slightly more expensive) model with a digital input?
No.
See this thread. Discussion of differences about half way through.
If small contrasty objects in the middle of your screen flutter between two adjacent pixels, it’s because synchronizing the analog values works badly enough that the pixel to pixel mapping changes over time. That problem would go away with a fully-digital connection.
I’ve heard several people swear DVI is fantastic. I haven’t had the chance to try it.