Draftwatch: 2016 Big Board Rankings Roundup

The NFL Draft is a game with some of the highest stakes in all of professional sports, and it's the ultimate market experiment. In the simplest terms "commodities" (the right to employ talented football players) are available for "sale" and teams pay (using draft choices) to obtain them. This means the value of a player (in terms of the NFL Draft) is set by what teams are willing to spend to "buy" the rights to any given player. The market price is ultimately determined by the buyers (NFL teams).

Using the actual results of the draft as a fixed baseline to determine draft value you can then work backwards and judge accuracy of various draft "big boards" that attempt to assign value rankings to players. These boards, created in the weeks and months leading up to the draft, are usually based on the opinions of their creators. Once the league decides what a player's true draft value is (by drafting him in a certain slot) we can compare that to how big board creators ranked that player and judge the accuracy of their assessments compared to the NFL's reality.

It's logical to ask why any of this matters at all and the answer is extremely simple: money. The draft is a massive business these days. Over the last 2 years (2015 & 2016) more than 1.8 million mock drafts have been completed by fans on the Fanspeak draft simulator alone. They are one of several options available (including sites like Draftwired.com and others) and that number will surely grow if current trends continue. Between 30,000 and 40,000 mock drafts were completed using the Fanspeak simulator on the last day before this year's real draft alone. America is crazy about the NFL and that unwavering devotion naturally drives massive attention to the mechanism that builds the league's player base (the NFL Draft).

Media outlets and ranking services that are deemed the most trustworthy and reliable will get the lion's share of the attention prior to the draft. That attention and viewer trust translates very directly into dollars. Viewership, clicks and marketshare in the lucrative pre-draft market all translate into higher ad revenues. The reality is that there's another, less visible high stakes game being played by the media behind the scenes of the already insanely high stakes NFL Draft itself; and the target is your eyes and ears. Hopefully this comparison will help you spend your valuable resources (time and energy) wisely when you are consuming pre-draft information next season.

Definitions

Big Board - A straight numerical ranking of all the players available in a draft. The highest rated players will be at the top (#1) and lower rated players at the bottom. Individual team needs or draft positions are not considered in this model.
Mock Draft - An educated guess about which teams will pick which players. Value of players is adjusted based on team needs, draft position and historical preferences.
For this exercise we ranked big boards, not mock drafts.

Methodology

This project was born when I was struck by how much variance there was in different big board's rankings of players and how much faith fans placed in these lists as a reflection of an actual player's draft worth. Given how many NFL front offices pay lip service to drafting the "best available player", you would expect there to be considerable overlap between big board rankings and what actually happens during the draft. Generally speaking, a "good board" (as defined by this exercise) should be able to roughly predict when a player is drafted.

I asked fellow WCG Contributor Josh Sunderbruch if he would be interested in helping me create a grading scale to rank the various big boards. Josh has a natural knack for working with numbers and producing terrific analytical content with the results. He was really excited by the potential of the project and hit the ground running. Over the next few weeks we hammered out trial grading scale, tested it on several previous year's big boards, and then fine tuned the scale until we were satisfied with it. The end result is a grading system that clearly indicates just how accurate a big board actually is compared to the real world results of the NFL Draft.

With the grading scale created we needed to select some boards to grade. I put out a call for final big boards from this season and collected ones that were published on the Internet. Meanwhile Josh did some digging and collected two prominent analysts' (Mike Mayock and Matt Miller) boards stretching back 5 years. This provided us with some historical depth to balance our grades and ensure this year's scores were not skewed by a single, atypical draft result. In addition to Mayock and Miller's offerings we scored the final 2016 player ranking boards from CBS Sports, Forecaster (compiled by Arif Hassan), Fanspeak, Chris Burke (SI.com), Draftek, NFL Mock (via Fanspeak.com), Optimum Scouting, Jeff Risdon (Real GM), and Lance Zierlein (NFL.com).

Finally, a frequent complaint is that the experts do no better than someone might manage throwing darts at a board, and we wanted to test that hypothesis, too. Before the draft Josh used a random number generator to assign a value to each player who was invited to the combine to create five distinct boards, each with their own random Top 100. Those were scored as well.

In order to reflect the belief that big boards should reflect what happens in the draft, we based our grading system on accuracy and defined that accuracy in terms of increments of draft rounds. A basic round in the NFL Draft has 32 choices, one for each team in the league. Simply getting near the right draft position (+/- 16 spots) was considered the neutral baseline. This means placing a player correctly within a 32-slot (one full round) range scored as zero points. For example, Chris Burke of Sports Illustrated placed Derrick Henry as the 30th best player in the draft, but he wasn't selected until #45. That's close enough to avoid a penalty, but not close enough to earn any credit. Burke received a neutral score for his ranking of Henry (0 points).

The entire goal of the project is to see how accurately experts project value, so greater accuracy results in an increased score:

• Placing a player within 8 spots (+/- 8 spots) of his actual draft position (or in the correct "half" of a round in draft terms) earned 1 full point
• Calling a player's value with more accuracy (within +/- 6 spots either way, or roughly one third of a round) earned 2 points
• Landing a player within +/- 4 spots (the correct quarter of a round) was worth 3 points.
• Predicting a player in his exact draft spot was worth a maximum score (5 points), because that's the kind of hole-in-one that people point to when defining success of pre-draft prognostications

While we rewarded big boards for their accuracy on the plus side we also wanted to include a penalty for large "misses" when predicting value. We wanted big misses to hurt a little, but not so much that they canceled out the positive accuracy results. You will see that misses are not worth nearly as much negatively as true "hits" are worth on the positive side.

• Missing by up to one full round (within +/- 32 spots in either direction, up or down) resulted in the loss of half a point (-0.5)
• Ranking a player incorrectly by more than a full round in any direction (more than +/- 33 slots in either direction) it was a "true miss" and a full point (-1) was removed from the overall score

For example, Matt Miller considered Hassan Ridgeway the 77th player in the draft; Ridgeway was drafted at #116. This was a big miss, so Miller was charged as such.

Full Grading Scale:

Exact match = 5 points
Match within +/- 4 slots = 3 points
Match within +/- 6 slots = 2 points
Match within +/- 8 slots = 1 point
Match within +/- 16 slots = 0 points
Match within +/- 32 slots = -0.5 point
Match within +/- 33 slots or more = -1 point

Results

So, how did the professional evaluators do? To know for sure we needed to check our random "control" boards. The five randomly-generated boards were total disasters, registering scores as low as -77.5 and as "high" as -56. They had an average of about 75 misses. The best random board missed by 33+ slots on 69 players, and the worst missed on 86 of them. This puts to rest the theory that throwing darts (random chance) is just as good a method to select football players as any other. Several NFL General Managers will be devastated by this news.

Mike Mayock swept the field. He is a juggernaut; the '85 Chicago Bears of the draft prognosticators if you will. Of the twenty-four boards evaluated, he had all five of the top five scores. His worst score (66 points for 2015) was better than the second-best board this year (CBS Sports, 64 points). For the 2016 draft alone, his score of 75.5 was three times as high as the average of the rest of the field, and his five-year average of 73.7 was nearly 10 points better than his closest competitor. Mayock clearly understands how the NFL tends to value players and is able to reflect that value amazingly well in his big board rankings year after year.

Matt Miller also racked up a consistently positive grade. His high score of 40 came this year (2016) and his low score of 20 occurred in 2012. He has shown steady improvement almost every year, indicating he studies his misses and learns from them. In total he earned a very respectable average score of 32 over the last 5 years.

Acknowledgements

I have the utmost professional respect for any analyst or outlet that takes the time to create a big board for the draft. It is a massive undertaking and entails a gratuitous amount of work. To then post that work with your name on it for all the world to see takes guts. They undoubtedly suffer the slings and arrows of the internet for their choices and random goofballs will repeatedly dredge up their "misses" in perpetuity for their trouble. That is some thanks for doing a boatload of work for the entertainment of the draft-fan masses.

Always remember that regardless of what you think of an outlet or analyst's work, that it is work and it's for your benefit. Certainly feel free to disagree with them (that's half the fun after all) but please do it respectfully. If you can't manage that then create your own big board, post it in the most prominent place you can think of and brace yourself for the backlash.

Many thanks to Fanspeak for creating the draft simulator in the first place and for sharing data with this project. Without their continual willingness to assist, this summary would have been much harder to complete and much less thorough.

2016 Big Board Scoring Summary

Owner	2016 Score	+/- 32 Miss	+/- 33 (or more) Miss
Mike Mayock	75.5	17	19
CBS Sports	64	20	22
Forecaster *	49	26	25
Matt Miller	40	24	25
Fanspeak	37	16	35
Draftek *	35.5	21	31
Lance Zierlein	9.5	21	41
NFL Mock	3.5	17	46
Chris Burke	1	20	41
Optimum Scouting	-1	18	43
Jeff Risdon	-7	16	54

* - Both Forecaster and Draftek are amalgamations of data from a multitude of sources to create ranked lists; not straight "big boards" per say.

Historical Big Board Scoring Summary

Owner	2016	2015	2014	2013	2012
Mike Mayock	75.5	66	66.5	75.5	85
Matt Miller	40	32	38.5	29.5	20

Ready to crown Mayock King of the draft?  Want to scream reasons from the ramparts why he is not?  Have at it in the comments section below.