Caloric needs, male vs. female, age differences

Mr. Athena and I have been trying to eat a bit healthier lately, both for weight loss and for general health reasons. We’ve noticed that he and I tend to eat around the same amount of food, which is a bit odd, given that he’s taller and weighs more than me. He’s also older.

I’m 5’6", 35 years old. Mr. Athena is 47, 5’10". We both work out, but he’s more consistent and does harder workouts than me.

Before we started eating healthier, I’d managed to gain about 10 to 15 pounds in the past 2.5 - 3 years, he gained a little more, maybe 15-20 pounds. On a given day, I just about always have breakfast, lunch, and dinner. He tends to eat only once between morning and dinner, but snacks both before and after dinner. He almost always ate a bowl of cereal before he went to bed. He’s got more of a sweet tooth than I - if there were cookies in the house, for example, he’d sit and eat half a pack in one sitting, whereas I rarely eat more than one or two cookies in a day. Meals, when we had them together, were typically very near the same size. He’d eat 3 eggs to my 2 in the morning, for example, or I’d give him the steak that was slightly (maybe an ounce or two) larger. Things like pizza or pasta would almost always be about the same size - 2 or 3 pieces of pizza each, or we’d share 1/2 box of dry pasta, cooked & sauced.
Before we started eating better, I just assumed he made up calories with his snacking and beer-drinking. Now that we’re watching what we eat, though, I know we eat almost the same amount (I tend to make meals, so it’s easy to gauge what goes into both our mouths.) We’re both losing weight - he a bit faster than I. I watch what he eats and think to myself “Hmm, I should cut portion sizes some.” But if I do, I get awfully hungry. He just seems to rarely get hungry at all.

I’m not overly worried about it, but I am curious. Why would a person my size and a person his size eat the same amount in a day and one of us not gain and/or lose a huge amount more than the other? Given that he’s bigger than me and works out more than me, how does this work? Does age and sex have that much to do with it?