Either way is gramatically correct. I would use “my children” if my spouse were not present or was not part of the discussion and “our children” if so.
Not necessarily present. If you’re just talking about your wife, it would be reasonable to use “our”, since it’s clear about to whom you are referring. The important thing in using pronouns is to be clear about to which person they are refrring.
I would use “our” if I had just recently mentioned my wife. “My wife and I went to the store yesterday. Our kids wanted ice cream,” sounds a lot better than “My wife and I went to the store yesterday. My kids wanted ice cream.”
NB: this isn’t so much an issue of grammar, per se, as one of sociolinguistic factors…
That being said, I agree with everyone else that “our” is appropriate first of all only if 1) the coworker has met/heard of your wife and it is clear from the context of the conversation that she’s in some way involved in what’s being said or 2) the coworker is a family friend who is used to encountering you as a couple. In fact, in this situation I’d say it’s pretty much necessary to use “our.”
I don’t believe that the answer hinges on whether the listener knows your better half or not so much as whether you mention her.
For example:
“I don’t let our children stay up later than ten” would be better as “I don’t let my children stay up later than ten” unless it was implict from previous conversation that your wife was in the picture.
On the other hand:
“My wife and I like to treat my children on their birthdays” would be out of place unless the children weren’t the fruits of your wife’s loins. You’d generally be better saying “My wife and I like to treat our children…”
In short, I believe the possessive pronoun used should match the subject of the sentence. I’m not an expert though - just a pedant.