I think what caused my confusion was the name. It led me to think that gravity was the cause of the acceleration, which isn’t the case. It’s the movement of the other object that causes your speed to increase and gravity is just the means by which you “transfer” that movement.
To use Munroe’s metaphor, if gravity was the cause then you’d get a bigger speed increase by bouncing a ball off a building than off a bus - a building has more mass than a bus. But it’s the movement of the bus that’s giving you the speed increase so a building is useless.