I thought he was saying:
“I know people will bring up the millers as an example of allegory. While it’s true that I never liked the looks of the Young miller, you can see I didn’t mean them as allegory because his father, the Old miller, had a black beard, and he was not named Sandyman.”