Well, what did you think of the earlier episode when Anthony Head was singing at a club in his dream when the gang was attacked by The original Slayer? That was one of those break-into song moments that you say you wouldn’t like.
My personal favorite moment will always be Spike trying to rush Buffy out of his crypt because he knows he’s going to sing and doesn’t want her to see it.
I say watch it–give it a shot. If you hate it, you can yell at all of us.
IIRC, that’s the only line Willow sings where she’s not in a chorus with someone else. She goes through entire songs where she just kinda stands there whilst everyone else sings, which goes along with the character’s established stage fright (also her guilt involving Tara, which is part of what makes “I’m Under Your Spell” such an eerily great song).
Also, I love how when they sing “and let it burn” in “Into the Fire” you see the fire truck go by in the background. 
Aly told Joss she really wasn’t comfortable singing, and he wrote her part with that in mind.
She also sings
“I’ve got a theory, some kid is dreaming,
And we’re all stuck inside his wacky Broadway nightmare.”
by herself.
Oh yeah, I forgot about that part
That’s even funnier if you remember her nightmare about being an opera singer from the first season episode with the kid dreaming all over the town. 
Honestly, given what you wrote in your OP, you’ll probably hate OMWF. I’ve noticed that people who dislike musicals as a genre outright tend to dislike the episode. It’s not surprising at all, as Joss’ love for the musical genre is made very clear in every moment, even when he’s mocking or outright parodying various musical cliches, and even though the whole thing is presented as a “meta” musical, with the characters fully cognizant of their singing and appropriately bewildered.
However, I was never a big fan of musicals either… I find sung lyrics to be hard to understand in most cases, which makes the whole endeavour pointless since the whole conceit is that characters burst into song at the most critical moments. And yet, OMWF is the episode that got me watching “Buffy” in the first place, and it remains one of my favorites. Joss was at the top of his game writing OMWF, as it combines the sharpest and most memorable dialogue of the last three seasons with genuinely catchy songs (an especially amazing feat considering that Joss had never really written music before). I’ve listened to the soundtrack hundreds of times, and it never gets old… did I mention that this is the only musical soundtrack I’ve ever bothered to purchase?
As others have mentioned, the cast’s vocals range from competent (Sarah Michelle Gellar, Nick Brendon, Michelle Trachtenberg) to excellent (Emma Caulfield) to absolutely mind-blowing (Anthony Stewart Head, Amber Benson). Alyson Hannigan is easily the weakest of the lot, but fortunately Joss wrote the musical with each cast member’s strengths and weaknesses in mind, and thus gives Willow very few sung lines but lots of good “actor” moments- reaction shots a specialty.
(Minor spoilers below)
It should also be noted that OMWF is easily the most quotable episode of “Buffy.” Other episodes will have a couple of great lines here and there, but I’ve heard references to virtually every single line in OMWF. Other people have listed a lot of memorable lines, but they’ve just scratched the surface. Nobody’s yet mentioned Anya’s rock out, Tara and Giles’s heart-breakingly gorgeous duet, Dawn’s reaction to the singing, or the awesome villain, performed by Broadway master Hilton Battle (the only demon on “Buffy” who can out-suave the leading vamps), or any of dozens other fantastic moments.
So yeah… I’d say, give it a shot, although don’t be surprised if you don’t like it. But if you do like it, even a little bit, be careful… it’s a grower, and it may quickly addict you. It may even turn your musical-phobia around for good. 
Gah. Damn coding.
Well, um, yeah, that’s kind of, you know, the point of the lyric. It’s a callback to that first season ep.
Damn you all. Now I have several of the songs from the episode playing “on rotation” in my head. Last time this happened I was humming “Under Your Spell” under my breath for weeks!
Hinton Battle. I’m glad they put the spotlight on him for a full song–he was well worth it.
Dawn: (petulantly) “I gave birth to a pterodactyl.”
Anya: (missing the point entirely) “Oh my God! Did it sing?”
Say, have you heard the Lord of the Rings musical based on OMWF? It’s called “Once More With Hobbits”, and is actually pretty good. Boromir’s version of “Let Me Rest in Peace” is my favorite song from that one.
One of my favorite musical bits is in Buffy’s “Give Me Something To Sing About” where every time she sings the word “Heaven” or Spike sings the word “Living”, they both go horribly flat, like they can hardly bear to sing it, much less sing it well. Also, I liked Dawn’s lead-in to her aborted musical number starting with “Does anybody notice? Does anybody care?” (she sounded like she was gonna have a REALLY good musical number before someone stuffed her into a sack).
Also, there are some good dance moments in there, especially with Michelle Trachtenburg’s two dance numbers at The Bronze towards the end. Did she do ballet at some point before she was an actress? Excellent dancer, makes running around in skittish fear look very graceful.
MT did study ballet as a child, and she has kept up with it somewhat, having used the skills in Ice Princess, on BtVS and on Six Feet Under.
Yeah, I’ve heard it, and it’s great.
They sing a diminished second, and though it sounds “wrong,” it’s exactly the notes Joss intended (check out the demo on the OMWF soundtrack). That particular interval is very dissonant and harsh to the ear, which of course matches the characters’ states of mind at the time.
IIRC, Michelle Trachtenberg has had dance training, and specifically asked Joss to give her a dance number rather than a major song.
And Tara does spirit fingers!
I’ve recently gotten the soundtrack, and I’ve found that I am pretty much incapable of not doing the hand motion when that line is sung. Which can lead to problems while driving.