In the week up to the NFL draft, we’re being bombarded by mock drafts and in depth analysis by draft experts such as Mel Kiper. The one thing I’ve wondered is anyone keeping these experts honest? Have there been any studies of the various draft experts and how accurate their predictions, both in terms of players being drafted and how well the players’ career turned out?
The difficulty in rating the prognosticators is a rather small sample size and the fact that the NFL draft is very heavily dependent on circumstances. One draft day trade could completely change the landscape of the entire draft, which nobody could really ever predict.
I’m sure the draft experts don’t want one.
Mock drafts can be fun, but they have no relationship to what really happens. Sort of like PECOTA in baseball.
The other problem is that most of the draft people do not always make clear distinctions between who they are saying a team *will *take or who a team should take.
If a prognosticator call player X a first-rounder, and he is indeed taken in the first round, but then sucks out loud, was said prognosticator wrong.
PECOTA is a falsifiable statistical prediction. Most of the things the draft people say are (wisely, I suppose) not really falsifiable.
The Huddle Report rates mock draft accuracy, both over a three-year window and year-over-year. Kiper is listed as #6 over the last three years and Todd McShay is #8, but McShay was better last year.
I’ve seen others, but I can’t seem to find them.
I often find Mel Kiper annoying, but I wouldn’t take any great pleasure in reading a web site that documented how often he’s been wrong in his assessments of various players. After all, Mel isn’t the only one who’d look bad if his goofs were publicized. Practically EVERY General Manager and pundit in football could be made to look stupid. That’s because assessing NFL talent is not an exact science, and no matter how smart/experienced you are, you’re going to guess wrong, a lot!
Mel Kiper didn’t have Tom Brady on his list of can’t-miss quarterbacks when he came out of college. But you know what? NOBODY did!
In 2001, the NFL’s top-rated quarterback was Kurt Warner, the leading receiver was Rod Smith, and the top rusher was Priest Holmes. What did they all have in common? NONE of them were drafted! Not a single team saw fit to spend even a sixth round pick on any of them.
So, Mel Kiper screwed up. He didn’t see stardom for any of those guys. But NOBODY did.
And it’s not because the NFL’s front offices and scouting departments are filled with idiots who don’t know anything about football. Most scouts and executives are highly intelligent, highly knowledgeable people. And they’re STILL spectacularly wrong in their judgments on a regular basis. Just like Mel Kiper.
Here is a good article from the Washington Post about Mel Kiper, Jr.