I have used that line recently, only to hear, “Ummmmm, I didn’t mention what day we were going”. 
And no, I wouldn’t waste a lunch on someone I wasn’t friends with.
If I wasn’t friendly with the person, I would probably only go if I had cash in my wallet (unlikely) and if it would extend my lunch hour past the time I usually get back (less likely because I usually take a later lunch). One of the things I hate about lunch meetings and birthday lunches and the like is that they always take place at noon, when the majority of my company goes to lunch. I cover my department during that time (except on special occasions) and go to lunch at one–getting back from lunch any earlier than two o’clock makes the afternoon interminable to me.
So yeah, I don’t think you did anything wrong. I don’t even think this was a “team player” situation. They weren’t going to lunch to brainstorm ideas for the new Whatchamacallit Project. If you don’t want to go, don’t go.
Going probably would have been the office politic thing to do, but if you chose not to go you really should have come up with a better excuse than saving money to spend on your boat. Doctor’s appointment, business meeting, gym, vague “other obligation”…pretty much anything would have been better than the excuse you gave.
Well, you have to use it judiciously. 
NOW you tell me!