Where did you see “the answer”? It could just as well be that the riddler waits until someone answers, and then, no matter what they say, says “No, you’re wrong, it’s actually _____”.
I Googled it and several cites came up with 16 as the “official” answer- and usually thought it was a trick question.
Trying to think about “mathematically” can get confusing. Not because you cannot formally define various things, but think about a torus. It has no boundary, okay, but does it have one hole? Or two holes (we are talking about a hollow torus, so you can also go all the way around the inside)? Or no holes (it would work as an inner tube and not let any air out)?
OK, but why though? I can’t see any reason to consider the collar a hole but not the bottom; you could pick any hole arbitrarily and call it the perimeter. There is nothing that distinguishes the bottom other than that it’s bigger.
Sure, pick any one arbitrarily to call the perimeter. And then the other three are holes. Whichever one you call the perimeter, the hole-count remains the same.
I think the important points have been made already, but, as usual, there’s an apropos xkcd:
Depends on your chosen frame, what a “hole” is. If a pair of pants is a sphere that has 3 holes, then a t-shirt is one that has 4. If, however, the pants are a disc with two holes, then a tee has 3.
Me, I think in 3-D, so I’m going to call it 4. Doesn’t answer the OP’s question, though, which I’m sure is a trick one.
It does kind of feel like, if we say that a sheet has zero holes, then we ought to say that a sphere has -1 holes.
This is why counting continuous boundaries is a better option IMO. An unbroken sphere has none, neither does an infinite plane. But a disk - a bounded part of a plane - has one, the outer boundary, just like a sphere with one hole. And every subsequent hole is a boundary within the disk, or on the sphere. The count stays the same for both. Pants or shirt, 3 or 4.
There are 6 holes that we know of.
The holes in the waist, sleeves and neck are implied, but could in fact be sewn shut. Furthermore, the entire shirt could be an elaborate fake, with the torso being a single sheet of cloth. So while there may be more holes, 6 is the only number we can be certain about.
So you’re counting the brownish spots as holes?
Of course, there real answer is “This is not a pipe shirt”.
The real question is “What color is this shirt?” I’ll bet there are some who think it’s red and not crimson.
Do the gaps between the atomic nuclei and the electron shells count as “holes”?
Exactly. If you imagine putting patches in all the holes, including the punched holes as well as the constructed neck, sleeve and hem openings, then the closed-up shirt is topologically equivalent to a sphere, and it had 16 holes in it.
But if you think of the shirt fabric as topologically equivalent to a disk, then the hem opening doesn’t count as a hole, but as the edge of the disk. Only the other 15 apertures are holes.
Correct. More generally though, you can consider any one of the holes to be the perimeter, leaving all the other 15 holes as holes. There’s nothing topologically privileged about the waist hem.
It’s common to think of the largest opening as the most “natural” one to choose for a perimeter. But that’s mostly because we’re used to the practical reality that sheets of real world materials are only so stretchy, and if you want a long edge, it’s best to start with the biggest one you already have.
Whereas the conceptual topological sheet material is infinitely stretchable with no elastic rebound. You stretch; it stays. It doesn’t tear either, no matter how far you stretch it. Handy stuff; I’d love to order a few yards of it for use around the house. 
Yup! Except of course anything else could infinitely stretch your conceptual topological sheet material too. You make a lovely patio awning out of it, and boom, a few flies land on top of your awning and now you have several fly-thickness pillars from the awning to the ground. 
Good point! 
I’d been assuming that there’s a sort of topological starch I could spray on my sheeting once I got it arranged the way I wanted it. At which point it became rigid. Or at least as rigid as I wanted it to be. However much that was.
Once we’re inventing magic stuff, let’s go all in.
Here is how I see it. No holes on a surface means any circle drawn on it can be continuously shrunk to a point. Two holes are equivalent If a hole drawn on one can be continuously moved to a hole around the other. So take a cylinder. It appears to have a hole at each end, but they are equivalent. Now take that cylinder and drill a hole somewhere in the middle. Now the holes at the ends of the cylinder are no longer equivalent. You cannot move a circle at one end to the other end; there is an obstruction in the middle. So it has three holes now; one at each end and one that you created. Following this reasoning, your basic T-shirt has 4 holes. Assuming the photo contains no hidden features, it follows that the answer is 16.
Incidentally, a torus has two holes. There is circle drawn through the hole and one drawn around the hole. There are other circles gotten by winding around the hole, but in the strict sense of algebraic topology they are made of from the two basic ones. No, I cannot go into details.
Are you certain about that? Wouldn’t it have two holes, and the third “hole” is really a perimeter?
That depends - if it was dyed with authentic kermes, then it’s crimson. Otherwise it’s just red.