I’m pretty sure it’s a Nat King Cole duet with a woman, and they’re singing about all the little, petty things that infuriate them about each other, but in the end, they really do love each other.
Lines that I’m pretty sure I remember verbatim:
“He can’t explain the lipstick on his shirt!” “That was cherry pie!”
“That’s the man every time, that’s the man…” sung together “We really love each other just the same!”
Can’t find a clip to verify it or anything, but sometimes the title is the hook, and you appear to have (almost) remembered it among the snatches of lyrics you’ve provided.
Could be “That’s a Man Every Time”
Check out the site Martha Tilton dot com slash discography dot html. Perhaps the forum on that site will lead you to someone with more information on the title.