Chicago Bears Number Mill: Trubisky’s Rookie Performance Associated with Favorable Odds of Future Success

Previously on Number Mill: Cam Meredith | Markus Wheaton / DVOA | Dontrelle Inman | Chiefs Offense | FA Receivers 1 | FA Receivers 2 | FA Edge Rushers / SAQ

Predicting the NFL is rough business. This time last year, plenty of analysts pegged the Giants as Super Bowl hopefuls, the Jets as a lock for the first overall pick, and Andrew Luck was unquestionably on track to start week one. When people use numbers to help predict NFL outcomes, they usually cite statistics that seem to support their opinion, but rarely attempt any analysis to determine the statistical significance of those numbers.

When people do attempt a more robust analysis, they most often report the correlation coefficient. I referenced some of these analyses in my edge rusher number mill. This is a valuable statistic for understanding how well a predictor (e.g. previous year’s sack totals) correlates with an outcome (e.g. future sack totals) but it’s hard for non-nerds to interpret its meaning.

Luckily, football analysts are not the only people who have put effort into developing statistical metrics for predicting outcomes and quantifying the effect of a predictor on an outcome. Today I will introduce a relatively simple metric used in medical research which does the later: the Odds Ratio.

The Odds Ratio tells you how much more likely someone with a predictor is to have an outcome. In the case I will present for rookie quarterbacks, you would simply take the odds of an average rookie quarterback turning into a franchise QB (let’s say 1:1 or 50% for a top 10 pick) multiply those odds by the odds ratio (in this case about 8) in QBs who are positive for the predictor, and get your new odds (now 8:1 or ~89%).

One nice thing about odds ratios is that if the predictors you use are independent, you can multiply them in series. If for example someone did an analysis showing that changing to an offensive-minded head coach in year two increased a QBs chances of becoming a franchise QB, it would be appropriate in Trubisky’s case to take those 8:1 odds and multiply them by the odds ratio for that new analysis. Fun!

For this analysis I chose to test whether good performance in a quarterbacks best 5 of their first 10 games would increase the odds of becoming a successful franchise quarterback. I chose this predictor after reading an article on this topic by Jonathan Wood earlier this off-season which showed that performance whole season averages of stats like passer rating, completion percentages, or TD:INT ratio did not reliably correlate with future success. The article also showed that QBs did not consistently improve as the season went on—some, like Carson Wentz, may have “hit a wall” or reverted to poor mechanics after the stress of the season wore on them. After reading Wood’s article, I decided to test for whether a QB showing the ability to play at a high level—whether that happened early or late in the season—would better predict future success than season averages.

The study population

For this analysis, I chose all quarterbacks drafted since 2004 who were either drafted in the first 50 picks or drafted later but started several seasons. I excluded anyone who started less than 10 games and anyone who’s outcome is yet to be determined (anyone drafted 2015 or later and some earlier draftees who’s career has been delayed by injury or a significant delay before starting).

The specific predictor

For my specific predictor (i.e. the measure of rookie success) I chose an average adjusted net yards per attempt (ANY/A) of 8.0 or greater for their best 5 of their first 10 games starting. I chose best 5 of the first 10 rather than the best 50% in their rookie season because several QBs I wanted to include in the analysis did not start many games in their rookie season or started an odd number of games in their rookie season. This way I was able to include players like Eli Manning and Aaron Rodgers.

I chose ANY/A because I believe it’s the best single unit that measures a quarterback’s effectiveness. Because it’s a per attempt figure, it isn’t affected by trends in pass % or play-calling preferences—things which say more about a quarterback’s surroundings than the quarterback himself. For those who aren’t familiar, ANY/A is essentially yards per attempt with additional yards added for touchdowns and subtracted for sacks and interceptions.

I chose a threshold of 8.0 after collecting data for the first two years of quarterbacks. My initial plan was to use a threshold of exceeding the 75th percentile for season average ANY/A, but I quickly saw that that threshold was too easy to pass. I prefer not to base my analysis parameters on the data that’s collected—this increases the chances of tailoring your model to noise in the data-set and getting a deceptively positive result—but in this case, I made the change with only a small percentage of data collected and it was clear my original plan was not going to work. It’s also important to note that I did not calculate Mitchell Trubisky’s result before deciding this threshold.

The outcome

The outcome I chose was “successful franchise quarterback.” I considered a few numerical criteria for defining this, but since this was only handful of people and it’s ultimately a subjective distinction, I decided to go fully subjective here. I identified eight “successful” franchise quarterbacks in the study population: Eli Manning, Philip Rivers, Ben Roethlisberger, Aaron Rodgers, Matt Ryan, Matthew Stafford, Cam Newton, and Russell Wilson.

You can argue that some of these quarterbacks don’t belong on this list, or that Alex Smith or maybe even Andy Dalton do, but a swap of one or two of these players wouldn’t have a large effect on the outcome (and in fact removing Eli or Stafford would make the predictor seem even stronger since they both failed to meet the threshold).

I left a number of notable players out of the analysis because I determined their status to be undecided including Andrew Luck (injury) Derek Carr (injury) Kirk Cousins (late starter and only successful in a single QB-friendly system).

The results

Surpassing an 8.0 average ANY/A in the best 5 of your first 10 games was associated with an 8.14-fold increase in the odds of becoming a successful franchise quarterback (p = 0.014). 6/8 franchise quarterbacks (75%) surpassed this threshold, while only 7/26 (26%) of those who did not become successful franchise quarterbacks passed this threshold.

The false-negative players (people who didn’t surpass this threshold but did become successful franchise quarterbacks) were Eli Manning and Matthew Stafford (hahahaha). The false-positives (those that surpassed the threshold but failed to become successful franchise QBs) include Kevin Kolb, Joe Flacco, Mark Sanchez, Tim Tebow, Robert Griffin III, Ryan Tannehill, and Geno Smith.

Mitchell Trubisky surpassed the threshold with an average ANY/A in the best 5 of his first 10 games of 9.98.

Full results for those included in the analysis can be seen in the table below.

Results for those determined undecided:

Two notable young quarterbacks who have not played 10 games, Deshaun Watson and Jimmy Garoppolo, have both already played 5 games at a high enough level to surpass the threshold. Their exact number can’t be calculated until they play 10 games, but in both cases, the minimum is above 8.

Out of curiosity, I modified the threshold to 8.75 to see how that would change the results. This removed three of the false positives, increased the odds ratio to 16.5 and improved the p value to 0.001. I believe this is more of a demonstration of the effect of tailoring your analysis to the data than it is evidence that the predictor is even stronger if you raise the threshold.

Discussion

Football is not a perfectly predictable sport. Any criteria used to predict success will have false positives and false negatives. But it’s still valuable to be able to say meeting a threshold is associated with an 8-fold increase in the odds of success. If we acknowledge that there are no certainties in football, but every outcome has a probability of occurring, we can benefit from tools that help quantify that probability and how much a statistic will effect that probability. The Odds Ratio is one of those tools, and I would love to see it get wider usage in the football analytics community.

In terms of this analysis, the most likely candidates to be false positives are volatile passers: gunslingers whose good days are great but whose bad days are too devastating to be sustainable franchise leaders. That might ultimately describe someone like Jameis Winston—who also meets the threshold—but its not a description of Mitchell Trubisky. A reasonable concern for the validity of this analysis is a high percentage of the newer (undecided) quarterbacks passing the threshold. The game of football is undoubtedly becoming more pass-oriented. I hoped that a change in trends would have less an effect on ANY/A since it is unaffected by number of pass attempts and a change towards increasing numbers of screens and short passes might actually decrease yard per attempt numbers, but perhaps the passing game has just become that much more efficient.

There are a lot of reasons to be excited for Mitchell Trubisky’s career moving forward. But it never hurts to add one more into the mix.