I’ve recently been on a binge of buying used PS2 games at Gamestop. Around the year 2000, I began exclusively playing PC games and pretty much sat out the entire gamecube/PS2/Xbox generation of console games. I really have no idea what were the great games for the PS2. Generally I like RPGs and strategy games, but I do have a weakness for the occasional rampage about and kill everything in sight genre. Also, FPS on console makes me queasy because I suck at using control sticks to aim, so those generally have to be really good for me to like them.
So far I own:
Katamari Damacy
Guitar Hero 1 & 2
God Of War 1 & 2
Manhunt
Resident Evil 4
Dynasty Warriors 4
Arc the Lad
SOCOM Navy Seals
So what games do I HAVE to play for the Playstation 2?
edit: I’m aware the entire Playstation 1 library is also open to me, but I played most of those back when I had a PS1.
Syphon Filter
Rainbow Six: Rogue Spear
Parasite Eve2
As someone who also skipped that entire generation and went back after, the stuff that stands above the rest of the library for my (heavily RPG and strategy favored) preferences are:
Persona 3 and 4 (best two RPGs and possibly straight out games on the PS2)
Final Fantasy 12 (bit of a change up from previous games, breath of fresh air for the series)
The God of War games, which you already have
A tier below that there are just way too many games to name - I wouldn’t call them MUST play but just so many games between “above average” and “very good” for that system. Suikoden 3 and 5, FF10, and DQ8 is a few hundred hours just to start with.
Ico
Shadow of the Colossus
Okami
Final Fantasy Tactics (still one of my fav. games to go back and play over on the PS)
Twister Metal 2 (it’s a classic game for the PS, and still kinda holds up for me- I could play this game over and over with my friends beating the bosses and all) It was the series that got me into buying a Franchise, but Twisted Metal one is meh, and 3 and 4 are just silly. It wasn’t until twisted metal black for the PS2 came along to give this game a worthy note to end on. Both of those are great fun games though. If they had made a new good TM game for the PS3, I’d consider it as well.
(Ah, I see you said you’ve done most of the PS1 stuff, so then nm on those- but I still say give TMBlack a Rental if you liked Twisted Metal 2, you’ll be pleased).
Also, Disgeae 1 and 2 are both GREAT games, I preferred two a bit more, but 1’s not bad. They’re great tactics games similar to FFT but with many new and interesting features and a nice sense of humor.
This triumvirate can’t be supported enough. Amazing and unique games.
More ideas :
Shadow Hearts 2 : Covenant is one of the best Jap’ style RPGs I’ve ever played. Amazing music, unique combat system, cool story (although you may miss out on some of it if you haven’t played the first, PS1 game)
Shin Megami Tensei : Nocturne is a stranger one. Extra weird story, odd rotoscoped graphics, and absolutely brutal gameplay. The combat is very, very hard. Which, of course, makes it great if you’re into punishment 
Final Fantasy X & XII are obviously missing from your list (the first is very formulaic, but extremely well made, the second is more MMORPG-like, but also more open. Both look gorgeous.)
Metal Gear Solid 3 is fantastic, but also somewhat hit or miss - you’ll either love it or hate it.
Prince of Persia : Sands of Time is none of the genres you mentioned, but it’s the one and only platformer I ever liked. It’s truely a masterpiece.
If you like fighting games, pick up any of the Soul Calibur series or the Tekken series. Those are must-haves for the fighting game fan.
I also really like Def Jam: Fight for New York-- it’s basically a bunch of rappers kicking the shit out of each other, and as silly as that sounds, the combat itself is actually a lot of fun. It’s martial arts and wrestling based, and the moves are insane. You can also grab stuff from around the ring to hit your opponent with (bottles, 2x4s, etc.), and the crowd will occasionally grab and hold a fighter for the other fighter to wail on. Unexpectedly fun, I must say.
Persona 3
Okami
Ico
Psychonauts : This is by far the most creative game I’ve every played. How am I the first to mention this?
Beyond Good and Evil
Grand Theft Auto Series
Fun crime based games with interesting cities built that you can explore as well as fun missions.
Kingdom Hearts 1 +2
Final Fantasy type (made by the same people from what I understand.) RPG game set in a very dispersive disney univerise. I found the main storyline to be convoluted, but the individual planet stories were always awesome. My favorite land was the Hundred Acre Woods.
Beyond Good & Evil: One of the best action-adventure games ever made. It plays kind of like Zelda, and yet something else entirely at the same time Great visuals and a good story make this one of the best games of the last generation.
Need for Speed: Hot Pursuit 2: A fantastic racing game and arguably the best in the series. I’m normally not a huge fan of racers, but I loved the racing/evading cops aspect.
Metal Gear Solid 3: This is the best Metal Gear game hands-down. Great gameplay, a decent (for once) story, and visuals that still look great.
Resident Evil 4: Good call on this one. However, while the PS2 version is competent, I urge you to play it either on GameCube or Wii for a superior experience. Even if not though, it’s probably still the best game available for PS2.
Yeah. I guess it is kind of dispersive. But, I meant to type immersive, which looking at isn’t even a word. I guess that is why spell check changed it to dispersive. Anyway, Kingdom Hearts is fantastic because you really feel like you are a part of a Disney environment on each world. You are immersed in it, if you will.
Toca Touring Cars
ISS Pro Evolution Football
Victory Boxing 2
Toshinden 2
Gran Turismo 3
At least one Tony Hawks skateboard game
Outlaw Golf 2
Some of these will be redundant, so I’ll try to keep the uber-top-list short:
Persona 4
Shadow of the Colossus
Okami
Metal Gear Solid 3
Resident Evil 4
God of War
And some more that I haven’t seen, yet:
Devil May Cry 3 - brilliant and brutal 3D action gameplay that is, IMHO, much tighter and stylish than God of War.
**
Odin Sphere - JRPG stylings meet 2D action brawler amid some of the best art ever in a game.
**
Sly Cooper 2 & 3 - one of the great platformers of the era. It has a comic book conceit to it, but the gameplay is a great blend of jumping around, acquiring new abilities, and stealthing your way into and out of areas. Imagine the freerunning of Prince of Persia put in the skin of a Mario or Crash game, and you’ll have an idea of what it feels like to play.
**
Jak 2 & 3** - another great character-platformer. This series stands out for its great use of vehicles, open-world goals, and simply unmatchable voice acting and character modeling for the frequent cutscenes.
**
Amplitude - the precursor to Guitar Hero. Plays a bit like GH with a controller, but you have multiple tracks to switch between, so you’re effectively remixing the electronica songs while you’re playing the game. Has powerups!
**
Yakuza 1 & 2 - A very Japanese take on the gangster genre of games. Not as open-world as most of them, but still interesting with context-based combat a ton of fully-fleshed minigames to explore when you’re not cracking heads.
If you like 2D Fighters, you’d have a hard time going wrong with Arcana Heart, otherwise, you can probably skip it. 
Seconding Persona 3, Persona 4, Okami, Odin Sphere, Shadow Hearts Covenant. Adding Tales of the Abyss, which I enjoyed a lot more than than any of the FF titles for the PS2.
Persona 3 and 4
Shadow Hearts: Covenant and From the New World
Ico and Shadow of the Colossus
Bully
the Warriors (one of the most underrated games of that generation)
Beyond Good and Evil
Psychonauts
Indigo Prophecy
Metal Gear Solid 2 and 3
Final fantasy X and XII
all of the GTA’s
Buuuuump…(not quite undead yet, I hope).
I just got a used PS2 to replace my dead DVD player in the bedroom, and figured I might check out a few games that I never got a chance to play, as I had the X-Box for the last generation. It came with God of War I, which I’m playing and really enjoy, and will look at many of the above.
But there’s a lot of suggestions for Ico. Ico is currently selling on eBay for ~$60! Is it that good - or would I be better off spending money on some of the other titles, including Shadow of the Collossus instead?
I haven’t played Ico, but from what people tell me, it’s not actually really much -like- Shadow of the Colossus. For me, that would be a good thing, because I found Shadow of the Colossus to be pretty but shallow. The story, such as it was, is told almost exclusively at the beginning and very end of the game, leaving the twelve or so hours in the middle (it’s also -short-) to be a largely featureless repetition of “Ride to location using annoying navigational method and useless map” followed by “crawl all over gigantic thing using vaguely frustrating controls while you kill it by trial and error”.
It’s evocative, but I found it fundamentally not much fun to actually play. Most people argue with me that this is because it’s art and I “don’t get it.” My only response is to roll my eyes.
That said, it’s not terrible, and for $20, it might be worth the gamble. But this tells you nothing about whether Ico is worth $60, and I’m afraid I can’t help you there.
Don’t miss Okami though.
Shadow of the Colossus was one of my favorite games on the PS2. It really is a singular experience which relies heavily on exploration and figuring out how to actually kill the colossi. Very atmospheric. Airk’s description though, is largely accurate. It works for some people, and not for others. That’s fine though.
I note that Odin Sphere is recommended above. Very well received game that I just could never get into. A 2D action/brawler/RPG hybrid with absolutely gorgeous art. It should have been right up my alley, and I tried really hard to give it a chance. I think I played it for about 10 hours before giving up on it, it just never hooked me. I’d recommend it to other people if it sounds interesting though, because it’s not a bad game.
ICO is a good game, but I don’t think it’s anywhere near $60 good. It does actually share some similarities with SOTC from a design perspective. You’re in a large, crumbling castle that’s mostly abandoned. There are “shadow creatures” that pop up from time that you have to fight, but it’s mostly focused on exploration and puzzle solving. The storyline is sparse and, again like SOTC, mostly unfolds in the first five minutes and last five minutes of the game. The reason it’s so expensive on eBay is because it’s considered something of a “cult classic”. There was a low initial run, it didn’t sell well, and there was never a “greatest hits” release. It’s very rare.
Ico is wonderfully atmospheric but it’s extremely short. I finished it in about five hours. On the other hand I got it for $5 so I wasn’t going to complain. $60 makes me wonder what those people are smoking; it’s not THAT rare…
If you like God of War, Airk, check out Godhand for a beat-'em-up that is a truly unique experience. Rather than be given the same set of moves that you repeat for twenty hours straight there’s a library of around one hundred and you construct your combos. And even the regular enemies are a solid threat to you as you learn the game system. It’s by the same studio that did Okami and they always did something quirky with a unique style (they also did the Viewtiful Joe platformers).
I have a feeling the people who are paying $60 for ICO and the like consider themselves collectors in addition to being gamers. There are a lot of older titles out there that are fetching ridiculous prices. Original, sealed copies of stuff like Xenogears and FF7 can go for upwards of a $100 on eBay.
Looking around I also found this listing which just made me shake my head…