You know that most advice isn’t worth the paper it’s written on (or the electrons it’s written with), so take this with a grain of salt and don’t blame me if it backfires on you. 
First of all, you could talk to the police and outline the situation to them and ask what in their experience would be the best way to handle the situation. Explain to them that you don’t want to file charges at this point. In fact, you don’t even need to tell them who the threat’s coming from; just outline the problem and see what they think. They may give you some idea of how to proceed, or they may insist that you file charges and then blow you off if you’re not willing to do that. Either way, you don’t have to have him charged in order to talk to the police.
Secondly, you might talk to a lawyer. Attorneys play hardball for a living, and a threatening letter from an attorney can often work wonders in taking the wind out of a person’s sails. Such a letter might be all you’d need to get him to give you a wide berth from then on.
Third, I’d consider what kind of beating I might be in for if he were to attack me. If he’s just a normal, everyday kind of guy who also happens to be outsized, then I’d think I might just be in for some bruises and maybe a broken nose. On the other hand, if he’s the kind of guy who hangs out in strip joints and does drugs and runs with a rough crowd, he might fuck you up in ways that will last the rest of your life - either cosmetically through facial scars and such, or physically through broken bones and/or damaged organs. In my younger bar-hopping and 3 a.m. breakfast days, I saw several guys get seriously fucked up by people like that, and I have no doubt the victims still see or feel the results of it to this very day. (The reason I’m telling you this is so that you don’t decide to call his bluff and take a beating unless you have to. That strategy could backfire if he’s truly a badass sort. Also I’d consider what you’re girlfriend’s family is like. If they’re fairly normal people, it isn’t likely that he will beat you viciously in front of them; on the other hand, if they’re white-trash druggie and alcoholic types, then he might.)
Then, failing that I didn’t get a handle on how to proceed by talking to the police and/or an attorney, I’d simply walk up to him (or call him on the phone if you don’t want to have the next family get-together hanging over your head) and tell him that: number one, you know he is a lot bigger than you and that he can kick the shit out of you anytime he wants, and therefore you have no intention of fighting him (this may mollify him because you’re acknowledging straight up that he can kick your ass, and thus he may feel less of a need to prove it), and that if he ever does attack you, you’re not going to fight back but you will file charges, and you will testify in court, and you will do everything you can to make sure that while you’re sitting at home eating pizza and drinking beer and watching movies on television, he’ll be sitting in a jail cell the size of his bathroom, and very likely sharing that space with a roommate who doesn’t like him very much. I’d also point out that at the very least he’s gonna be out thousands of dollars in bail and lawyer fees. And then I’d ask him if he really thought beating me up was worth all that.
Then I’d explain to him that you like his family (unless this is blatantly untrue and he knows it) and you have no reason to try to drive a wedge between your girlfriend and them, but that sometimes for one reason or another you simply aren’t able to make it to family get-togethers. Then I’d suggest that everyone let bygones be bygones and there’s no reason to take things personally in the event that the two of you don’t show up on occasion. On the other hand, if there’s mutual dislike between her family and you, then I’d simply say let’s let bygones be bygones and all try to get along the best we can.
In any event I would probably get a small can of pepper spray* and carry it in my pocket anytime I’d be likely to encounter him. It probably wouldn’t hurt for your girlfriend to carry one too so she could come to your aid if necessary. (Plus, if you’ve already talked to the police, that would count in your favor should you zap him with the spray and he tries to claim that you attacked him.)
At any rate, I wish you the best of luck in getting this situation resolved.
- Be sure to keep your head about you if you ever use pepper spray. Don’t spray against the wind, and don’t spray where it could fall back into your own face - otherwise then you’ll really be helpless.