I don’t think this is completely off-base. Little Drummer Boy falls into a Christmas song uncanny valley – it’s not as fun or jaunty or catchy as the more festive songs, but it’s not as pretty or spiritual or moving as the best carols.
I personally hate it because it’s one of the few I can’t “tune out.” Songs with repetitive rhythmic lyrics get on my nerves generally, and Little Drummer Boy is a prime example of the genre.