On Ryan Pace's Drafting Process

Kamil Krzaczynski-USA TODAY Sports

Recently, there have been some good articles on WCG talking about Ryan Pace's performance as a GM. Specifically, his ability as a drafter. I particularly liked this article by Josh Sunderbruch.

I wanted to go in a slightly different direction. With this piece, I'm not so much interested in how good Pace's picks themselves have been. I want to look at how he's approached the draft as a whole in relation to his peers around the league.

Because sometimes, I think we can get too hung up on results to the exclusion of process. And fair enough, it's a results driven sport, a results driven league. But when you're trying to evaluate objectively, I think you have to acknowledge that good process can sometimes produce bad results and vice versa. Especially over a shorter timeline.

So I wanted to take a look at Ryan Pace's process as a drafter.

Fair warning, I'm going to spend some time talking about my methodology, because a lot of the numbers I end up working with are of my own creation. If you don't care so much about the nuts and bolts, feel free to scroll down to the table titled "2015 - 2021 NFL DRAFTS (plus already-made 2022 pick trades)". I'll even put a header saying 'STOP SCROLLING IF SKIPPING METHODOLOGY' to let you know where it is.

Evaluating Draft Power

My objective with all this was to look at every NFL team and get a sense for how they use their Draft Capital. In other words - Based on how much draft power they receive, how do they spend it? And when I say 'spend it', I mean how they use it before any picks are made.

Not every team gets the same amount of value to start each draft. A team with the #1 pick (and the 33 pick, the 65 pick, etc) has considerable more power than the team with the 32nd, 64th, 96th, etc. The biggest driver of draft power is record the previous season, with worse teams getting more value to spend. And then comp picks come into play as well.

But the draft power a team receives isn't necessarily what they use by the time they're picking. Teams make trades all the time, both pick-for-pick on draft day, pick for player swaps, and hybrid picks/players trades. I wanted to get a sense for how each NFL franchise is using that draft power.

In order to do what I wanted to do here, I needed a numerical value I could assign to each pick in the NFL draft. These numbers in theory correspond to how 'good' a given pick is. I didn't find anything out there I liked a bunch. So, I set out to create those values.

To build that table of pick values, I used PFF season grade and playing time data, for each season between 2006 and 2020. This allowed me to look at the entire 4 season span of each draft class's rookie contract length from 2006 to 2017.

Wait, Isn't PFF Terrible/Nonsense/Fatally Flawed?

I know PFF has bad reputation around here, at least in some circles. After a cursory look, I can understand why some people dislike it and what their valid critiques are.

But I will say this - I aggregated the PFF data for all players over a span of 15 seasons and did some analysis on it, and I think these grades have some collective merit. I won't go into the details, as that would take an article at least as long as this one just on that subject.

Very short summary: I took PFF grades, weighted them for playing time, and compared them at a team-wide level with both more 'objective' measures like points scored and yards gained (for offense) and points conceded and yards conceded (for defense). I also compared them to team offensive/defensive DVOA, as I think that's a metric that people have some respect for.

While I wouldn't say PFF grades match up perfectly, I think there's enough of a correlation that it's fair to say that OVERALL, these grades correspond to reality to the extent that this exercise is potentially worthwhile.

Am I saying take every PFF grade for a player/season as gospel? Certainly not. Are there players that PFF gets wrong on a consistent/systemic basis? Very possibly. Are there flaws with PFF's methodology that result in inaccuracies in the grading? That too seems possible.

But for the purposes of what I'm doing, these grades don't need to be perfect. As long as they're broadly accurate enough that they roughly mirror the reality of performance.

If you're of the belief that PFF grades are complete noise/garbage, though, this analysis won't do anything for you. And it's probably a good idea to X this tab out and save yourself the time.

Classifying Each Player/Season

PFF ranks each player on a variety of measures, but each offensive and defensive player gets an overall Offense Grade or Defense Grade. For those unfamiliar, these grades are supposed to represent their performance over the season, evaluated on play-by-play basis. A good play gets 'plus points' and a bad play gets 'minus points', and the grade reflects the balance of play. 60 represents a neutral grade, with 95 being about the highest you'll see and anything below about 40 being extremely bad.

I used those overall Offense or Defense grades.

One problem right off the bat with PFF grades is that they're in no way weighted for playing time. A guy could play every snap on defense and get a 75 grade. Another guy could play 50 snaps and get an 80 grade. Obviously, just looking at raw grades alone will have issues.

So along with the grades, I gathered playing time data for all player seasons. I then benchmarked each player's snaps (or whatever playing time metric made most sense for the position) against either their own team's total off/def snaps or the league average, and classified each player/season as either a qualified starter, a significant contributor, or not enough snaps to classify as a significant contributor that season.

Then, I placed each player's performance (by PFF grade) into one of 9 buckets. The top bucket was for players who scored highest in relation to all qualified starters at their position that season. The second bucket represented a slightly lower grade but still much higher than average peer grades. And so on, down to the last bucket representing far below graded performance relative to positional peers.

Finally, I gathered each player's draft status (I was only looking at drafted players) and whether a given season was within the first four seasons of being drafted.

Player Season Criteria

Now I had a table of every player who played on offense or defense between the seasons of 2006 and 2020. I have a 1-9 bucketed performance ranking for that player season. Whether that player played enough to be considered a starter, a contributor, or didn't play enough to be considered a major contributor that season.

Here's an example of what the table looks like, only imagine 25,000 rows instead of 18. I picked a Bears player/season and a player/season people might be familiar with from another team, to give some idea of what this looks like.

PS Examples

(Note that the Performance Rankings are relative to a player's peers at a position. So for instance you can see Cody Whitehair 2018 with a slightly lower grade than Kyle Long 2016, but a higher PR bucket. This means that starting guards in 2016 had a higher average ranking than centers in 2018)

Now, it should be said that these buckets aren't meant to be analyzed on a player/season to player/season basis. It is definitely fair to quibble about whether a player was truly '1-Elite' or '2-Excellent' in a given season, etc. But again, in aggregate for thousands of player/seasons, I think they're relatively accurate.

Oh, also - I left out Kickers, Punters and Fullbacks in all this. Just too much extra hassle for not enough impact on the overall results.

Adjusting for position

The second major problem with these PFF grades when it comes to trying to evaluate the draft is that PFF grades aren't position dependent. In other words, a QB gets a grade from ~30-95. A TE gets a grade from ~30-95. The best tight end season is 95. The best QB season is 95. But we know that QBs are considerably more valuable.

So we need a little bit more context in assigning value to each of these player/seasons.

I created a positional adjustment table that attempts to give more credit to seasons from players at a more valuable position.

To make this table, I used a combination of two sources:

First, the site Over The Cap created a table based on differences in player salaries.

Second, PFF created a table based on a version of their player WAR.

I wasn't able to say that either approach was clearly better, so I weighted them equally and came up with this:

Positional Adjustment

I can say right now that I think the QB adjustment should probably actually be higher. I was OK being kind of conservative for this here. But if I were trying to refine this, I suspect that QB number goes up.

It was kind of interesting to me that CB and safety ended up this high. But I don't necessarily disbelieve that. Secondary play seems to matter more and more in the modern game.

Now each player/season also has a modifier score. Meaning that a QB season within a given bucket counts for 6.1 while a HB season in a bucket counts for 1.

Pairing pick numbers with player/season performances

I then gathered data for each draft pick number. For pick number #1, I looked for every season (within the first 4 years of being drafted) where a player picked #1 had played enough snaps to qualify as either a starter or contributor. Each of those seasons falls into one of the nine performance buckets, from Elite to Worst.

I tabulate all those seasons for each pick number.

each pick

(Sorry this is so hideous - the real thing has more columns and it was helpful for me to have sharp color dividing lines)

To be clear - What I'm tabulating is not players at each pick #. It's player/seasons. So for instance, at pick number 10, take Patrick Mahomes.

  • He was drafted in 2017, and he only had 44 dropbacks/rushing attempts. Not enough to qualify. So his 2017 shows up nowhere here.
  • In 2018, he qualified as a starter, and PFF grade put him into the 1-Elite bucket for QBs that season.
  • In 2019, again starter, this time he ended up in the 3-Very Good bucket.
  • And in 2020, again starter, again in the 1-Elite bucket.
In each case, this is a quarterback season, so 6.1 positional adjustment. So the #10 pick gets 12.2 into the starter 1-Elite, and 6.1 into the starter 3-Very Good.

Now for every pick, we have a sum total of how many seasons of performance those picks produced at each quality level.

Creating pick value scores

I combined the data in two ways. First, I did a top 3 score combination, looking at how many either 1-Elite, 2-Excellent or 3-Very Good seasons were produced. Those are kind of like the home runs, the Pro Bowl level seasons that every team is desperate for.

Then I also combined the T6 performance levels. Basically, this represents getting a season of at least relatively decent performance. I give no credit for season ranked at 7-Poor down to 9-Worst. While these players were on the field, their play was probably bad enough that they were hurting their team.

Finally, I made each pick # into a rolling 5 pick average. My reasoning for this is that I don't think any of us believe there's something inherently magical about the #27 overall pick. Like it's even possible for #27 to be better than #26. But over the 12 drafts analyzed, it's pretty bumpy in ways that seem unlikely to be anything but noise.

Here's a charting of T3 and T6 player/season values by draft pick #. (Pick number is the bottom axis)

Pick Values Chart

These values happen to correspond pretty well with a logarithmic trendline/equation.

I took that equation and fed each pick number into it, producing these values:

Pick Value Table

Like with the positional adjustments, there are some things I'm a little dicey about here. For one thing, I'm pretty confident that specifically the first few picks should have higher values.

Another thing that was just a little too complicated for me to mess with was the fact that first round picks have the fifth-year option. There probably needs to be an extra bump for every first round pick just for that.

If I had more time, I'd also like to maybe try to incorporate a salary based component. A fifth round pick that hits is going to be paid peanuts compared to a top 10 overall pick, and that value should probably be reflected.

But again, I was OK being relatively conservative here for these purposes.

Determining initial draft power for each team during the Ryan Pace era

I then grabbed the original draft order for every team from 2015 to 2021. This happens to be the span during which Ryan Pace has been the GM of the Bears.

Using the Pick Value table, I calculated the number value of 'Draft Power' each franchise has had over the past seven drafts.

I then grabbed all those drafts and looked at the picks each team has actually made, and calculated the number value of the picks each team has used on players over this time frame.

Finally, a number of teams have already traded away 2022 draft picks. To simplify my process and save time, I only looked at picks traded from rounds 1-3 (so for instance, the Bears aren't penalized here for not having their 2022 4th rounder). And debited teams who are missing 2022 draft power and credited teams with extra 2022 draft power. I projected draft order based on Week 16 W/L records, so the final values will change a bit in all likelihood.


Here's how the NFL shakes out over Ryan Pace's tenure as GM:

Draft Power Table No Players

By raw numbers, the Bears have used the least amount of draft power relative to how much draft power their Win/Loss record gave them. By percentage, they're just ahead of the Rams.

Now, maybe the first thing you'd think is 'Hey, what about the Khalil Mack trade?' The Bears gave up picks for a big-time player, so of course that's going to drop them down by a measure like this. And that's true.

For reference, here's that same table, only with each team and every trade they've made where they gave up significant (3rd or higher) picks to get a player. In parentheses by each player's name is the plus/minus draft power for that trade.

Draft Power Player List

So out of the Bears' -302, -106 comes from the Khalil Mack trade. A big part for sure, but even if you gave the Bears 'back' the 106, they'd still be one of the most wasteful teams.

The list of players traded for/away helps you get a sense of what certain teams/GMs have been doing.

For example, you can clearly see what the Rams have been up to. Trading huge amounts of draft power away for a collection of quality to star level players.

The Jets look bad. While they're 90 to the plus side of the ledger, look at the amount of NFL talent they've traded away to get there.

Similarly, the Raiders are high in this table, but when you take a look at the players they've traded away, it's less impressive. The Jaguars are a similar story.

You can also see why the NFL world absolutely trashed Bill O'Brien before he got fired from Houston. Woeful draft power return for DeAndre Hopkins. Wild overpay for Laremy Tunsil, especially given that they traded Duane Brown for a third of that. Just a trainwreck from an asset management standpoint.

Teams like Cleveland and Baltimore are interesting because they've accrued significant draft power without trading lots of contributing veteran players. And like with a lot of aspects of the NFL, you could probably write an article just about what the Patriots have been doing.

An Important Caveat

I do want to make this point - None of this analysis is looking at what teams are actually doing with the picks they make. It's entirely possible for the Texans with their 998 points of draft power to draft a better pool of players than the Browns with their 1858 points.

So in itself, the above table isn't the last word on who's doing a good job or a bad job. The table shows which teams are valuing their draft capital based on the history of what a given pick produces, and which teams are not.

But back to the Bears

I think this should be an area where we look at Ryan Pace with serious skepticism.

Because here's the thing - If we're debating whether Ryan Pace should remain Bears GM, we care about the future of what he's going to do. What he's done so far has already happened. Nothing can change the picks/trades/FA signings/etc he's made.

Which means the biggest question is whether Pace has a process that is likely to produce good results in the future.

I think this analysis here shows there's a flaw in Pace's process. Relative to pretty much the entire rest of the league, Ryan Pace doesn't value his draft capital.

It is possible for Ryan Pace to make up for not valuing draft capital properly and still be a good GM. He could do so by making such good picks that he nets out OK. In fact, if you were defending Ryan Pace you'd say his process requires him to seemingly not value draft capital. He's able to find a small number of players who are going to exceed their draft position, and he needs to aggressively trade up for those players when the time comes.

Whether that's actually happened or not, I think the evidence is at least clear that he's not pulling the wool over the entire league's eyes. You may or may not like Ryan Pace's draft picks, but I think it's already clear that he doesn't have some kind of magic secret sauce.

Whenever there's a group of organizations and one of them sticks way out from the pack, I think you need to take a close look and ask what's going on. There's nothing inherently wrong with doing things differently. In fact, sometimes the one doing things differently is the smartest one, the one who's figured out how to exploit an inefficiency in the group think.

But I don't think that's what's going on here with the Bears. I think we have a GM who is making mistakes in his overall process, and it's decreasing the chances of him constructing a contending team.

A final note

I know things with this fanbase get contentious, and the Ryan Pace issue has been a big flashpoint. And that there are a lot of partisans on either side of the issue, with some pretty dug-in positions.

I can promise you that this analysis I did, for all its faults, was fair. I didn't start with a conclusion and work numbers to reach what I was looking for. I honestly had no idea how the final numbers would come out. Obviously I had suspicions, but I was surprised to see the Bears were quite as bad as they were.

With that being said, I could still just be wrong about this! I may have made some errors in my methodology. I'm not a statistician, so maybe I made some bad choices that led to invalid or misleading conclusions. I will say - I did go ahead and try some different positional adjustment values and tinkered around with some of the pick values and reran the final table. The Bears continue to be pretty much exactly in the same place with regard to Initial Draft Capital vs. Actual Used Draft Capital. Unless my numbers or model is WAY off, there's probably something legit to this.

This Fanpost was written by a Windy City Gridiron member, and does not necessarily reflect the ideas or opinions of its staff or community.