So I want to design an accessible website...

I did go ahead and implement the ems as opposed to px measurements in my CSS, and the enlarge/reduce text works MUCH better now.

I’ll add that to my list of things to do to make it accessible when developing new sites. Out with pixels, in with ems.