One of the most commonly cited metrics for evaluating quarterback performance is completion percentage: the number of completions divided by the number of attempts. The blatant problem with completion percentage is that it does not account for the fact that not every pass attempt is equally difficult or valuable.
Imagine a situation in which it’s 3rd down and the offense has 20 yards to go. The quarterback elects to throw a checkdown at the line of scrimmage to their running back, who gets tackled for a meager gain of two yards. The offense is forced to punt the football away. The quarterback contributed little1 to their team’s chances of winning with that play and the pass was one that any NFL quarterback could make. Yet that pass boosted their completion percentage.
The solution would be to determine that the probability of that completion being made is quite high, so it isn’t particularly impressive that the quarterback completed the pass. And if the completion probability for every pass attempt could be estimated, we could also estimate any given quarterback’s expected completion percentage and compare it to their actual completion percentage for a better representation of their accuracy.
This is not a novel idea, of course. The NFL’s Next Gen Stats hosts a metric called “Expected Completion Percentage” on their site that takes into account variables such as passer speed, time to throw, target separation, and pass rusher separation. This data is not available to the public, unfortunately, so the point of this project is to create a model for completion probability using the limited data that is freely available.
The first task is to actually get the data. I used a play-by-play data set that consists of every regular season play dating back to 2009.
So, what features should be used for the model? The first answer that comes to mind is the distance of the pass attempt, or the number of air yards. Passes that travel a further distance tend to be completed at a lower rate. I plotted this relationship below — note that the size of each point represents the total number of attempts that traveled a certain number of air yards.
Passes don’t typically travel further than 20 yards from the line of scrimmage. The most common distance for a pass attempt is just five air yards. The mean pass distance is approximately 8.43 air yards, while the median distance is just six air yards.
Our model is gonna need more features than just distance, though. What about the down of the play? I calculated expected completion percentage (a preliminary version of it, not what will be the final product) using just the number of air yards. Then I compared this with the actual completion percentage on every down to see if there’s a difference:
If the down of a play did not have an impact on completion probability, one would expect the difference between expected and actual completion percentage to be the same for every down. But this is clearly not the case. Completion percentage on 3rd and 4th downs is far lower than expectation based on the number of air yards alone. Why? My guess is that defenses expect a pass on those downs so are better equipped to stop it. Since 2009, quarterbacks have dropped back on 46.5% of 1st down plays versus a whopping 74.7% rate on 3rd down.
That leads us to another potential feature for the model: yards to go. The defense probably isn’t going to set up the same way on a 3rd-and-1 as they would for a 3rd-and-15. Let’s compare the expected (still based solely on a regression of air yards) and actual completion percentages on the number of yards to go for each down.
The error (difference between expected and actual completion percentage) on 1st and 2nd down is consistently low for basically yards to go values.2 However, there is clearly a noticeable level of variance in the error on 3rd down, suggesting that the number of yards to go is indeed a factor that impacts completion probability.
What other factors impact the passing game? It becomes easier to defend a passing attack as the field gets shorter, i.e. as the offense moves closer to the opposing end zone. When you’re only 20 yards away from scoring a touchdown, there isn’t as much field to work with. Your receivers can’t space the field as much, so to speak. A graph of the expected and actual completion percentages at every spot on the field clearly demonstrates this phenomenon.
A model based on just air yards is pretty accurate until the offense is approximately 25 yards away from scoring. It plummets in goal-to-go situations, where the average completion percentage is just 50.1%.
There are plenty of other features that I ended up putting in the model that I won’t analyze in as much detail. Among these are the score differential (the difference in score impacts the probability of a pass play being called), whether the play is in shotgun formation, whether the play is in no-huddle formation, and the pass location (left, middle, or right), whether the play occurred outdoors or in a dome, the wind speed, etc.
Anyway, let’s get to the results.
The line that runs through the graph is not the line of best fit. It is what I like to call the “expectation curve.” A quarterback that falls on that line has a completion percentage equal to their expected completion percentage — perfectly average accuracy. Anything above the line is above average accuracy, and anything below the line is below average accuracy. Also, the size of each point is representative of the number of passes they attempted. Only quarterbacks with at least 900 passes3 thrown since 2009 were included in the graph.
The most accurate quarterback of all-time is unsurprisingly head and shoulders above his peers. Yes, even after including a feature in the model that identifies whether a pass attempt occurred outdoors or in a dome. While Brees has certainly benefited from playing in a dome for most of his career, it does not discredit his unprecedented precision as a passer.
Deshaun Watson has the second-highest completion percentage over expectation, but his sample size is obviously quite small, as shown by the size of his point. It is also rather sad that the data only goes as far back as 2009, because I’m curious to see how Peyton Manning would stack up with the rest if more of his peak seasons with the Indianapolis Colts were included (2004 especially).
If people were asked to guess who the far left point represented, I think Jameis Winston would be a popular answer. You don’t put up a TD-INT ratio of 33-30 in a single-season without being the absolute definition of a gunslinger. Credit to him for maintaining a completion percentage above the low expected rate, though.
Also, Mahomes and Rodgers share basically identical expected and actual completion percentages. It lends even more credence to the comparisons frequently made between the two passers.
In general, the points below the line appear to be smaller than the points above it. After all, the Blaine Gabberts of the world aren’t good enough to keep a starting job for very long.
I also calculated single-season completion percentage over expectation (CPOE) and found that Brees’ 2018 season was the best since 2009 in terms of accuracy. While his expected completion percentage of 63.6% was high, his actual completion percentage of 74.1% blew it out of the water. Russell Wilson (2019) and Tony Romo (2014) followed him up with stellar seasons of their own that didn’t warrant a Most Valuable Player award.
One last thing: I think it’s a shame that this data only goes as far back as 2009. Therefore, I created a separate model for completion probability that dates back to 2006. The caveat is that it includes far less variables and is less accurate, but it’s a bit better than just ignoring three years of football. Fortunately, the most important feature by far (which is confirmed by the feature weights for our first model), air yards, is available dating back to 2006.
All of this data is available under the site’s navigation menu. I have labeled this model that dates back to 2006 as a “basic” version of completion probability, while the model that dates back to 2009 is “advanced” due to its additional features.
In case you’re lazy, the links below will direct you to the pages with the full data:
Here’s a sneak peek: Drew Brees owns three of the top four most accurate single-season passing performances since 2006.4 And four of the top-six. And five of the top-seven. Once again, the most accurate quarterback of all-time. But you don’t need a model to see that.
- You could argue that in certain situations, the quarterback helped their team by not taking an unnecessary risk that could’ve been more costly than a punt. But I don’t think the ability to not throw an interception on 3rd-and-20 is worthy of praise.
- The clear exception is first down plays in which the offense has less than five yards to go. These plays rarely ever actually occur, so I don’t think that’s particularly noteworthy.
- This is admittedly a completely cherrypicked threshold for the sole purpose of including Patrick Mahomes.
- Minimum 300 attempts