Making him contact you is exactly what I was going to suggest. From there you can judge how he’s going to act around you. And from there you can lay the ground rules on what you would let him do.
As for your mom, I’d like to add a slight extra thing you might do. Now, this is just an idea from someone with no experience in this area, so take it with a grain of salt. But sometimes an outsider perspective helps you make clearer decisions. I know I tend to like input from everyone, as long as it isn’t mean.
That extra thing is, the next time she calls, empathize with her. Say something like “I know you want us to all get together and be a happy family. But it’s really up to him now. He’s got to reach out. He’s got to show that he won’t be mean to his granddaughter like he was to me.”