Could he be chewing it off himself? I have a high-strung cat that one day had bald spots on his front legs and belly. The vet said that it was a classic example of neurotic licking. When he’s feeling good the fur grows back in; when he’s upset about something suddenly he’s patchy again.
It’s the part about some fur growing back in already that makes me think it might be the problem. Fur grows just as slowly as our hair - it takes several weeks for Sam’s fur to grow back. So if it’s already coming back in one spot, that fur’s been gone longer than three days.