I am a human geographer who also works with data on land use and land cover and other, more “physical” things.
Math was always my weak link, from grade school to today. I don’t exactly have math “phobia”, but what comes more easily to many has always been a struggle for me. But at least I do appreciate what math offers (in my profession and beyond), and enjoy the minor victories of finally figuring certain (what many would consider simple) things out.
A few geographers get by – even thrive – with no math at all. These would be the most abstract human theorists. But 99 percent of human geographers, and 100 percent of physical ones (and almost all of both have some training in, and use of, the other), use some math, at least.
Statistics, for sure, is necessary. You don’t need to be an expert in matrix algebra or even multivariate analysis, but you’d better know how basic things like chi-squared and Pearson’s correlation work, and when you should use them. A bit of geography-soecific stats (clustering, spatial autocorrelation, etc.) is good to know about, too. Even if you just do more anthropologically-oriented work (including “political ecology”), you’re surely going to work with census data, questionnaire results, etc.
Then, there’s GIS, which is just a set of tools, nothing to be scared of (though anything with computers is frustrating sometimes). You don’t necessarily need much math skill for GIS, depending on your tasks. Some familiarity with logical syntax is helpful when making attribute-based queries. Many tasks are best done using raster cells of values, and then you are talking about math operations (simple sums, Euclidean allocation, etc.), bit I didn’t find this to be too difficult to grasp, even though, again, anything math related is a struggle for me.
In sum, if you truly cannot imagine dealing with a bit of math ever, then geography is not such a good option. But if you’re willing and able to dip your toes in just a few basic math things, there are many threads within the discipline that are open to you.