When Analytics Go Bad

Regular readers know that I am a fan of analytics, and that I tend to think that there is a real advantage to applying statistical analysis and advanced metrics to the discussion and practice of football. However, there are a number of reasons analytics have yet to catch on as seriously in the NFL as they have in other sports. Some of those reasons have to do with the limitations of an analytic approach to the small sample size of a 16-game season (yes, I know it’s now a 17-game season, but we don’t have any data from that season, yet). Some of those reasons have to do with how difficult it is to isolate factors in a team-driven game like football. And yes, some of them have to do with stubbornness and an old-guard mentality.

A lot of it, though, has to do with the pure ignorance with which analytics is sometimes applied to football. One such piece is a recent bit of nonsense by Josh Hermsmeyer. I’ve read other things by Hermsmeyer, and he seems like a reasonably intelligent person. As such, and in deference to the piece’s home on 538 (which also carries political content), I am going to paraphrase William F. Buckley, jr and say that I don’t want to insult his intelligence by suggesting that he really believes what he wrote.

I hope he doesn’t.

I’m linking it here in an effort to be intellectually honest, but I feel the need to attach a warning as if I am the principal from Billy Madison. Read it at your own peril.

Hermsmeyer’s premise is to use the second (non-rookie) contracts that positions earn as a way of determining the value of positions. From this, he assigns a relative value to each position, and you know what…I can’t. Here’s the actual explanation:

“To quantify how valuable quarterbacks are relative to other positions, we did something obvious: We looked at what they’re paid. Using data provided by Jason Fitzgerald at Over the Cap, we took the salary information for all non-rookie contracts signed since 2014 and expressed each player’s average salary per year as a share of the NFL salary cap. Next we calculated the median contract value at each position, ranked the positions, and finally expressed each as a share of QB value.”

Get that? Context doesn’t matter. So, the fact that his own data says that only two fullbacks received non-rookie contracts compared to twenty-two linebackers is not taken into consideration, despite the fact that not every team will even offer contracts to fullbacks (there are 22 active fullbacks in Over the Cap’s database) and every team needs at least a couple of linebackers (the league has 236, per OTC), the fullback is valued more than the linebacker. This is a sampling error the likes of which I would not allow in my English 101 class, let alone one that any self-respecting statistician should allow. This is a sampling error so bad, with cause and effect so poorly misunderstood, that I expected a footnote remarking on the Chicago Bears having a tendency to draft players who liked to wear orange and dark blue.

Of course the fullbacks who actually receive second contracts are going to get paid! They are the ones who have shown that they deserve to be paid and not replaced with a 6^th-rounder. Meanwhile, of course the median value of linebackers is going to be dragged down—there are so many of them that it’s not an apples-to-apples market. Consider that Hermsmeyer tells us that 30 cornerbacks were offered non-rookie contracts from 2014-2020, and they are all the same to him. Sure. Shutdown corners like Jalen Ramsey should absolutely be put in the same grouping as aging special teamers like Sherrick McManis, despite the fact that Hermsmeyer divides left and right tackle from one another.

Based on this sort of truly misguided grouping, Hermsmeyer concludes that drafting a quarterback is the only smart thing to do with a first-round pick. He has no qualifiers for context, only a minor concession on sample size that is breezily brushed over. At one point, he even says:

“The larger point is clear, though: QBs are king. In fact, quarterbacks are so valuable that if you’re a GM who’s not in a position to draft one in the first round, your job changes dramatically. When all the viable first-round QBs are off the board, your task effectively becomes one of loss minimization. Rather than attempting to pick the right players — which is a largely futile exercise anyway — or focusing on need or best player available, GMs can most effectively increase the expected value of their picks by selecting players at the highest value non-QB positions. If you can’t increase your hit rate through better evaluation, it’s best to simply take your shots at the most valuable positions and hope you get lucky.”

This is such an insultingly poor proclamation that it’s hard to wrap my head around it. Picking the “right” players is not a futile exercise just because it’s not always a perfectly matched scenario. To be clear—Hermsmeyer is actively saying that drafting a left tackle (any left tackle, apparently) is better than drafting a wide receiver (any wide receiver), and that drafting any wide receiver is better than drafting a linebacker. I am going to give him the benefit of the doubt and assume that he meant to imply that these were only scouted players and players with some sort of minimum grade on them, but he never says that. He never applies a minimum threshold, until the very end when he says “any” first-round quarterback is the best decision.

By Hermsmeyer’s analysis, drafting Sam Darnold, Josh Rosen, and Mitchell Trubisky were all good moves. They were better moves than drafting Denzel Ward or Quenton Nelson (Darnold), Minkah Fitzpatrick or Tremaine Edmunds (Rosen), or Jamal Adams or T.J. Watt (Trubisky). I am not exaggerating. Here is a quotation from the article itself:

“What’s notable about this analysis is that it doesn’t matter if the players turn out to be good. White was in the running for Super Bowl LV MVP, but that doesn’t mean choosing him at fifth overall represents a good process. Middle linebackers — even the best of the best — simply aren’t valuable enough to spend that much draft capital to acquire. The median fullback and punter earns more in his second contract than off-ball linebackers, yet teams would likely never spend a fifth overall pick on either position.

Even a celebrated pick of an NFL-ready athlete is probably a mistake when that player isn’t a quarterback. Kyle Pitts may end up being a generational talent at tight end, but the odds are quite poor that he will be able to return more value to the Falcons than if they had selected a QB with the fourth overall pick.”

Say what? Leave aside that the use of second contracts this way is already such a poor move in sampling and math that it boggles the mind. Just focus on the fact that he does not think it matters how a good a player turns out to be. Getting a good player on a price-controlled contract is a bad idea if it’s not a quarterback, because the quarterback is always the better move. His own chart, titled “We Can Tell Right Away Where Drafts Went Wrong” demonstrates the error of his process, though.

Among his “bad” decisions, he includes drafting Roquan Smith in 2018 as the 7^th-worst move since 2010. Huh. Wasn’t Mitchell Trubisky on a price-controlled contract already that season? In fact, isn’t that the season right before Trubisky played his best football? So, to be clear, Hermsmeyer’s argument is that drafting Devin White or Roquan Smith is a “value vacuum” (his actual term), even for teams that already drafted quarterbacks and are benefitting from already having a first-round quarterback on a price-control contract.

Would-be defenders of his can’t even tell me that he didn’t mean that—he includes Roquan Smith in his chart as the seventh-worst first-round pick, when in 2018 there was every statistical reason to think that Trubisky was going to remain the starting quarterback for the duration of his contract.

Okay. Trubisky was bad.

What about the value vacuum of the Cardinals drafting Simmons, also on his chart, one spot later? This is also the season immediately following Arizona drafting Kyler Murray.

In fact, three of his worst thirty moves were made by the Bucs, including two (White and Vea) that helped to build the defense that just helped secure a Super Bowl win. Yes, obviously, Tom Brady helped immensely. However, that same defense helped the Bucs land Tom Brady.

This is the problem with Hermsmeyer’s claims. There is no allowance in his analysis anywhere for the existing contract situation, existing roster construction, or even the multiplicity of roles on the field. Apparently, teams should just draft first-round quarterbacks in perpetuity. Simply put, Hermsmeyer is like the figurative physicist using spherical cows to increase milk production.

Or, to put it more blandly, this sort of statistical analysis isn’t useless. It is worse that useless. It undermines the craft. The gives a bad name to those who want to apply logic and thought to a situation. Hermsmeyer should be embarrassed. I’m embarrassed for him.

I tend to think Ryan Pace has been a bad GM for his tenure with Chicago. However, on his worst day, he is lightyears ahead of an armchair GM like Hermsmeyer. This goes back to another point, and it’s one that statisticians and armchair analysts (mea culpa, as I am guilty of it sometimes too) so often forget.

Finding talent is not luck.

I am one of Ryan Pace’s harshest critics. I would also bet almost any reasonable amount that Ryan Pace on his worst three-year stretch could do better than simply randomly assembling picks from Hermsmeyer’s value triage list and hoping to “get lucky.” When attempting to draft a player, a GM must assess scheme fit, character, work ethic, athletic talent, and roster needs. Drafting a quarterback when there is a promising rookie on the roster is a poor move. For that matter, drafting a generational talent at guard when he is available is a great move, even if on his second contract he is going to earn less than a medium-level edge rusher, precisely because that reduces the role of luck. I’ll put it in plainer terms:

Imagine that there is a guard who has a 99% chance of being great and being worth a second contract, with that 1% leftover being the chance he is a bust. Now imagine that there is an edge rusher who only has a 33% chance of being great and being worth a second contract, with the other 67% chance being that he is a bust. Who should I draft with my first-round pick?

If you think you have the answer, you’ve already made one of the same mistakes as Hermsmeyer.

You need to know who else I have on my team and what my salary commitments are. If I have 30% of my cap locked up for three years in edge rushers and no interior line, the answer is a lot clearer. On the other hand, if I have All-Pros on my interior line and have a vacuum at edge rusher, then the answer is a lot fuzzier. It also matters, though, what kind of edge rusher is available.

Most of the analysis that calls the draft luck overlooks the fact that GMs are narrowing a field of thousands down to only a few hundred. Then, of those few hundred, they must narrow them down to a few dozen that might or might not fit on the team in question. If they get only the tenth-best player available instead of the second-best, most metrics call that a failure. This is hard work. It’s an amazing display of skill that they ever get close to right. And this leaves aside the reality that players do not develop in a vacuum either, and just like Mike Martz had no use for Greg Olsen, Lovie Smith found the perfect use for Devin Hester.

Is drafting an off-the-ball linebacker with a high pick a bad move? Contract-based analysis will apparently say it is. However, that same analysis says the Bears failed at drafting Roquan Smith and should instead have taken (in order) Josh Rosen, Isaiah Wynn, Billy Price, and Daniel Carlson.

Analytics are wonderful when they are applied with logic as a way of furthering intelligent conversation. When they are used as a way of attaching numeric value to nonsense, they are mere pseudoscience masquerading as informed thought.