Because the custom is definite. There is a single, specific behavior associated with visiting. It’s similar to the way you’d say “what’s the procedure when visiting someone’s house?”
If you were to rephrase, you could say either “Bringing gifts to someone’s house is an American custom,” because other groups have other customs, or “bringing gifts to someone’s house is the American custom,” because there’s one specific custom in America.
Neither is wrong. In this situation, I would use the definite article because you are talking about a specific custom to do with being invited to someone’s house - it is the custom in this circumstance. I would be more apt to use “a” if speaking of one custom out of a whole range of them.
In the context, I see a difference in meaning. If you tell somebody “it is the custom,” you’re saying you must do this to avoid opprobrium. “It is a custom” is more lenient; choosing to do this would fit with tradition.
I’d agree with that. To me, one implies an almost mandatory behaviour (in America, it is the custom to tip wait staff), the other implies an optional behaviour that would be appreciated (in the UK, it is customary to buy a bartender a drink occasionally if you are a regular).