If they were pestering me for answer, I would probably get pissed off, but I don’t think I would go any farther than to say “I’m sorry, I don’t know why, I’m just not attracted to you in that way.”
The only similarly analogous situation I’ve had in my life came in college. There was this girl who was (and still is) a very good friend of mine. Attractive, sweet, intelligent, kind. On paper, there is absolutely no reason why I should not have been attracted to her. However, I always felt our relationship as more familial than romantic. We’ve shared the same bed, I’ve driven her car down to New Orleans (where she’s from) and stayed at her folks house for a few days when she had a summer internship in California (I think it was.) I even ended up photographing, many years later, her wedding.
One day my senior year, I came home to an answering machine message with her, basically, proclaiming her love for me. I was absolutely thrown for a loop. Had no clue whatsoever. After chain smoking a few cigarettes, I called her and basically explained the whole thing. It felt really weird. The whole theme basically came down to the old “I love you as a friend, but I just don’t feel a romantic attraction for you,” hopefully couched in diplomatic terms. After all, this was a person I loved as a friend, but I honestly just didn’t see her “in that way” for whatever reason. Hell, I didn’t know why I didn’t see her in that way, just that I didn’t feel the spark or romantic attraction. Attraction is a funny thing–we can’t really control it, but I do think it’s important in a relationship.
Thankfully, being the person that she is, she was mature and accepting of my reasons, and didn’t press the issue. We’ve remained friends since, as I said above. If pressed, I would have simply stood my ground and continued along the lines of “I’m really sorry, I just don’t see you that way, and I can’t do anything about it.” It sucks, as the other person, when you don’t have control of the situation, but it was the truth, and I honestly didn’t have any answers beyond that, and certainly didn’t want to hurt her purposefully in any way.
The experience has made me a lot more empathetic to the fickleness of attraction, and if I hear a line along the lines of “I just don’t see you that way,” I completely understand and don’t necessarily think it’s just a standard blow-off line, even if it might be.