What Do The Draft Experts Know? Tracking the 2012 Draft

I want to begin with the simple observation that it is not possible to really evaluate NFL Draft experts on their opinions of prospects. If there is some highly touted player who fails, then it is too easy to explain that the circumstances were outside of that player’s control. Likewise, if some player prospers when he was considered a marginal prospect, then it is simply too easy to say “well, goes to show what the right system will do for a kid.” Add in how difficult it is to measure one player’s impact on a team game, and the best we can do is create rough estimates of rough estimates.

With that disclaimer out of the way, it is possible to create a rough sketch of draft experts after a few years. For this exercise, we’re looking at the 2012 Draft through a specific kind of draft boards—the “best of position” board. Two familiar draft experts are going head-to-head: Matt Miller and Mike Mayock. I compared these position rankings to the scores each player recorded in Pro Football Reference’s Weighted Career Approximate Value, and I checked to see if they really did identify the top players in each position.

As long as all we want is for the experts to correctly identify the top five players (regardless of order), they do fairly well.

Looking at the top five in each position, across offensive skill positions, Mayock and Miller both put in a respectable showing. Their biggest shared misses came from underestimating Russell Wilson (both preferred Weeden) and from not considering T.Y. Hilton (both favored Blackmon).

For added benefit, it’s worth pointing out that Miller identified Kirk Cousins as the biggest sleeper QB and called Justin Blackmon the biggest potential bust at wide receiver.

Moving to the offensive line, things get a little fuzzier. For example, both men had Kelechi Osemele as one of their top-rated guards, even though he ended up at tackle. Likewise, Mayock had Cordy Glenn as a tackle instead of a guard. Ultimately, however, there were more hits than misses here, too.

Both men thought too highly of Philip Blake (who was drafted but never played) and Jonathan Martin, but they did well in calling value. Beyond both projecting that David DeCastro would be the best guard from the class (and according to PFR, they were right), they also were in-line, in terms of projections, with about when these players were drafted.

Over on the defensive side, things get much, much shakier. Obviously, there is difficulty even tracking or assigning position sometimes. Still, even with generous allowances made, 2012 was not a year for draft experts (or even front offices) to show their skill. Both men identified only 3 of 5 top defensive tackle prospects, and they made the same mistake as one another. They failed to appreciate Derek Wolfe and Mike Daniels, instead overvaluing Jerel Worthy and Devon Still. They had a high opinion of Shea McClellin (they also both thought he would play outside linebacker, however).

Broadening the field to simply “front seven”, then of 20 possible correct calls, Mayock had 11 correct calls whereas Miller only had 9 (and this is using generous definitions, as Miller thought highly of Bobby Wagner the outside linebacker). As a bonus, Miller felt that Luke Kuechly had the highest bust potential.

Finally, there are the defensive backs. Here, Mayock only offers 5 safeties total whereas Miller offers five of each type. While Mayock correctly identifies 7 of 10, Miller correctly identifies 8 of 15. Both missed Janoris Jenkins and Josh Norman, with Miller calling the former of those the “biggest risk” among his position group.

Mayock ends up with a score of 42 hits when 65 were possible. Miller has 41 hits when 70 were possible. Most of the time, when they missed front offices agreed with them.

When the rate of injury is taken into consideration in the NFL, and when the ability of scheme, coaching, and opportunity are so variable, this is really an amazing success rate. They are not so reliable as to be foolproof, but they do seem to do a good job of getting a general sense of how prospects will do.

Just as a cross-check, I looked at the same experts’ Top 100 boards and compared those lists to Pro Football Reference’s CAV overall. What was interesting was that while many, many players were on all three lists, fully one quarter of the fifty-plus players with a CAV at 20+ (the defensive Brandon Marshall/Trumaine Johnson level) were not in either expert’s Top 100. Additionally, three of the top ten players from that draft (when it comes to CAV) were not on either expert’s Top 100 list. There were nearly as many players who both experts thought were elite prospects (listed among the top forty overall by both and in the top three of their position) who never reached that level. To repeat, nearly a quarter of the “top” guys ended up outside of the top half of their own draft class.

Thus, while the experts were able to (generally) identify the top prospects, there was a lot of variation. This is important to keep in mind when the draft happens and people think “wait, they drafted that guy?” or “that’s way too early for him.”

On Thursday remember that both boards had Shea McClellin (14 for Miller and 31 for Mayock) above Jared Crick and Bobby Wagner (or T.Y. Hilton and Russell Wilson for that matter).