I remember 84 a little bit, but I really remember watching 88 (I was 8 years old) and not being able to get enough of it. I always liked the Winter ones better, but I hate that they split the years now so we have the Olympics all the damned time. I wish it was like it used to be when it was a big deal.
I also still miss the Cold War. 
1980 Lake Placid Games. I had just turned 8. The family was at the cabin in Maine for the February holidays, and my parents had brought the color TV (something they’d never done) just to watch the games. I remember Eric Heiden and the US hockey team. I also remember thinking Tai Babilonia was a sandwich meat.
I enjoyed it so much that I was crushed when the US announced it was boycotting the Summer Games.
Tommie Smith and Black Power in 68.
I remember watching Peggy Fleming skate and Jean Claude Killy ski in 1968. I was in Mexico for the 1968 summer (well, fall, really) games.
We had tickets to some of the less popular sports (field hockey, water polo, fencing, rowing, can’t remember what else). Also got to see some of the diving practice sessions. Also got to see some of the track and field finals (that was unplanned; I think my aunt and uncle surprised us with tickets for most of the family). I’m not sure exactly what we saw, but I think we saw Dick Fosbury in the high jump. (I was not quite 11.) Mom got to go to the closing ceremonies with her best friend. I remember some of the venues better than the events I saw there. I can still see the pool where we watched the water polo match.
We watched as much as we could on TV and were always watching out for Olympic athletes wandering around the city.
I vaguely remember some of the 1968 Olympics. In particular, I recall that the U.S. men’s basketball team was weak and vulnerable, since many of the best black American college players (like Lew Alcindor) were boycotting the Olympics, as a political statement.
As a result, the Russians looked like they had a chance to beat the Americans at “our” sport.
Spencer Haywood came through and won the game and the gold for the U.S., making him a very popular man in my house.
As with a few others in this thread – the 1972 Munich Games. I was 9 years old then (actually, I must have “tuned in” to TV that year as far as remembering things of note, because I remember the general election coverage later on that year too). Olga Korbut, Mark Spitz, and the terrorist attack – that’s what comes up from my mental filing cabinet.
I was oblivious to the Olympics until my Mom took me to see Tokyo Olympiad when I was 11 years old. I was mesmerized. Unfortunately, viewing the games has become less compelling over the years. Too many commercials, and too much sap .
Tokyo 1964. I was in a swim club at the time and was fascinated by Dawn Fraser winning a third consecutive gold in the 100m and Kevin Berry and Ian O’Brien who had won gold medals later turned up to one of our training sessions and let us all wear their gold medals. Betty Cuthbert who had won the 100m and 200m gold medals in 1956 came out of retirement to win the 400m gold medal. I don’t recall actually seeing much of the action though.
I can recall a fair bit of stuff that I actually witnessed from Mexico City in 1968 - Bob Beamon, Dick Fosbury, the men’s 200m and the Black Power salute, and Mike Wenden won the 100m and 200m freestyle breaking the World Record while still at Uni. Wenden was a one-off, swimming like a paddle wheeler. He used to take 50% more strokes per lap than other swimmers.
1994:
(Okay, that happened before the Olympics, but yeah, that year is the first clear memories I have of them.)