The problem is that I don’t think you can save the changes back to the video file that way.
That said, handbrake does have an option to increase the volume, as shown in this video:
I’m not sure what’s going wrong for the OP: do make sure you don’t just “pass through” the audio.
Also, you’ll unfortunately have to experiment with how much to increase the gain, as Handbrake doesn’t currently have a normalize function.