Well, I think that Fluxx is fun for the whole family. If you don’t believe me, just ask Cleophus, DocCathode, iampunha, danceswithcats, Siege, Ariel (sp?), etc.
From my childhood, when we were very young (hands really too small to hold the cards) war & fish. (and even later, both of those made good time filling, non-thinking card games.)
Once we could actually come close to holding a hand of cards:
uno, spoons, black jack, speed, crazy-8s, spades, hearts, variations of spades, whist, bridge, rummy, gin rummy, (we tried canasta once - no one really liked it), and of course, Poker … lots of poker.
BTW, “Set” (you do have to buy special cards for it) is a great game - I highly recommend it.
I agree with the Mille Bornes and Uno recommendations. My family started playing Mill Bornes when one of my older brothers had ear surgery one summer - he couldn’t do any swimming or sailing the rest of the summer, and he hated that.
We all started playing and became highly competitive with it. We still play holidays when we’re all together, and it’s just vicious. Now that we’re all older, we no longer throw the cards at each other when we get angry.
Really, it’s a lot of fun. It does take some basic math skills, but since it’s played in teams, if there’s one adult and one child on each team the adult can add what needs to be added.
Uno’s fun to, although my family turns that into a competition as well. 
Boggle is fun, but youngsters playing against adults are at an obvious disadvantage. When my daughter was small, we had a Big Boggle game, and we gave her a handicap: The grownups had to make words of 4 letters or more; she was allowed 3 letter words. Also we gave her one point more for the various word lengths than the adults. This made it possible for her to actually have a shot at doing well without us having to “let her win.”
She was (and is) just as competitive as we are in games, and at first would throw a fit when she lost, even in a no-skill game like Candyland. I asked her if what she wanted was to win. She said yes. Fine. Moved her playing piece to the winner’s spot. Hurrah! You won! Want to win again? Repeat. No, no, that’s not what I want. I want to play the game. Oh, that’s different. But you won’t always win. End of problem.
Poker is fun for around 8 years old and up. Also, Bullsh*t is fun. We call it ‘Bullcrap’ around here.
Spoons is probably the easiest game for the very young to get. I think most 3 year olds can play it.
Old maid, crazy 8’s and Uno are good too.
For non-card games, some that I’ve been successful with w/ my 4 yo are:
Yahtzee junior
Krawall im stall (uproar in the barnyard) is a good game. (i think i spelled the German wrong)
20 questions
I spy (if you can take it)