It’s been years since statistics for me too, but if I read your OP correctly you are not adding your 4 dice together, so we need not concern ourselves with a bell-curve distribution of the sums.
If his die is a 1 (1/20 chance), then each of your dice has a 19/20 chance to be higher.
If his die is a 2 (1/20 chance), then each of your dice has an 18/20 chance to be higher.
.
.
.
If his die is a 19 (1/20 chance), then each of your dice has a 1/20 chance to be higher.
If his die is a 20 (1/20 chance), then each of your dice has a 0/20 chance to be higher.
Once you have done this for all possible single-die rolls, multiply each of the final probabilities resulting from his throw by 1/20, then add all of them together for the total probability that at least one of your dice will be higher than any throw of his. I get 0.475, or 9.5/20.
If I’m full of it, a more prob-oriented Doper will no doubt inform us. If I’m not, they will no doubt tell you of a much simpler formula that has been lost to my memory. But I think this works.
In the case where you add 5 to his result, it’s just a matter of accounting for that in the calculations.
If his total is a 6 (1+5), each of your dice has a 14/20 chance to exceed it, etc.
Anything he rolls over a 14 has no chance of being beaten.
I get 0.2625, or 5.25/20.