by Brandon Ally, S2 Cognition
PT 2: Cognitive Skills That Separate the Best QBs From the Rest
HIGHLIGHTS:
In 27 NFL starting QBs tested, S2 measures of Tracking Capacity and Impulse Control accounted for 51.2% of their completion percentage
In these same QBs, Impulse Control accounted for 24.2% of interception rate
Magic formula for QBs is S2 Overall above 85, with very high Tracking Capacity, Impulse Control, and Distraction Control
One of the most intriguing takeaways from our Part 1 analysis of NFL QBs was that the S2 Overall Score was a strong predictor of Career Passer Rating. Specifically, 29% of the difference in Career Passer Ratings among the 117 NFL QBs we evaluated was accounted for by their S2 Overall Score. For example, if two QBs have a 30-point difference in Passer Rating (e.g., 98 vs. 68), then 10 of those points are attributable to differences in their S2 Overall Score. The S2 Overall Score is comprised of 9 specific cognitive skills, so we were naturally curious about how the specific cognitive skills related to specific on-field performance statistics. The findings from Part 1 were at a global level, which inspired further examination of associations between distinct domains of cognitive functioning and specific in-game performance metrics of the 27 starting QBs in this Part 2.
Modern analysis of NFL QBs understandably centers on decision-making. However, analysis is typically constrained to the outcomes and results of decision-making rather than processes and components of decision-making. For example, it may be tempting to conclude that QB X makes a lot of good decisions or is an effective decision-maker because he has high completion percentage, low turnovers, and a high number of TDs. However, to definitively say that QB X is a good decision-maker, we really need to know something about how he makes decisions. Otherwise, we might conclude he is a good decision-maker when other factors might be behind the performance success – he has a stellar offensive line, gifted WRs who make things happen after the catch, a talented RB who carries the load and opens up the passing attack, or operates in an offensive scheme that prioritizes high probability of successful passes. Moreover, because a play results in success doesn't mean the QB made the best decision. We've heard coaches and front office executives talk about big plays that resulted in scores, etc., and say, "yeah, but it wasn't the right decision for the situation!" When the only measure of decision-making is the outcome of a play, there is risk of circular thinking – concluding a good decision because the play was successful while also saying the play was successful because of a good decision. The only way to really know how a quarterback makes decisions is to measure the QBs decision-making process or skill. It is tough, if not impossible, to record all of the thoughts, processing, and reads a QB makes in real-time. Alternatively, measuring a QB's capacity for specific decision-making skills and determining how these abilities are associated with on-field success can provide unique insight into what decision-making skills are critical to success. The S2 Evaluation examines the cognitive processes that go into on-field decisions. By having this information, scouts, recruiters, coaches, and front office personnel can know what type of decision-maker the athlete is without having to guess by watching on-field performance or retrospectively codifying decisions based on the result.
One of our objectives at S2 is to unpack how split-second decisions are made, examining both the speed and accuracy of decisions in compressed time and space with millisecond precision. It's critical that decisions are evaluated within the context of complex in-game competition.
Diving into the data
One of our primary analytics tools for digging deeper into data is multiple linear regression, which offers flexible methods for modeling the relationship between a metric of interest (e.g., on-field performance) and one or more explanatory variables that have potential to predict that metric of interest (e.g., distinct cognitive processes measured by specialized S2 tasks). Linear regression has many practical applications, in this case, predicting behavior or explaining the variation in behavior due to the explanatory variables. To follow up on the linear regression models, we also highlight some correlation analyses provided by the regression. It should be noted that a correlation shows the strength of the relationship between two variables but does not show causality between the variables. To allow the reader to focus on the substance and not be distracted by statistics in the text, we included all stats (i.e., r-values, p-values, and F statistics) in a Results section below the article. Let's dig in!
Predicting completion percentage
The first step in the process was to run a stepwise linear regression with the dependent variable being completion percentage and the independent variables being the 9 isolated cognitive skills in the S2 QB evaluation: Perception Speed, Search Efficiency, Tracking Capacity, Visual Learning, Instinctive Learning, Decision Complexity, Impulse Control, Distraction Control, and Improvisation. In other words, we wanted to see which, if any, of the cognitive processes measured by the S2 Evaluation predicted completion percentage and to what extent. The regression model excluded 7 of the 9 cognitive processes due to lack of significance, leaving Tracking Capacity and Impulse Control in the model. The biggest contributor to completion percentage was Tracking Capacity, or the ability to broaden attention to see across the entire landscape and anticipate where defenders and receivers will move across the field1. The model revealed that 28.6% of the variation in completion percentage between QBs is due to Tracking Capacity. When adding Impulse Control to the model, the predictive effect was increased significantly2. Impulse Control, or the ability to play controlled and patient while making relatively few impulsive mental mistakes (forcing throws before he's fully analyzed the situation), contributed to 22.6% of the variation in completion percentage. When added together, Tracking Capacity and Impulse Control predict 51.2% of a QB's completion percentage. Bottom line: for a QB to achieve a high completion percentage in the NFL, their decision-making systems must allow them to see player movements across the entire field and to have exceptional control over reflexive impulses that can lead to premature or forced actions.
In academic peer-reviewed scientific publications, only p-values of .05 or smaller are allowed to be reported as significant. This means that if you tested your hypothesis 100 times, you would find the same, significant result 95 out of the 100 times. It is somewhat of an arbitrary cutoff, suggesting that something that happens 94 out of 100 times isn't worth reporting. We say this because 3 other cognitive processes contributed meaningful explanatory power in predicting completion percentage but fell just below the threshold of p<.05 and wouldn't be deemed significant by the more rigorous standards in academic publications. One of these cognitive processes was Instinctive Learning, or the ability to diagnose unscripted looks and pick up on subtle and less obvious tendencies of the defense and adapt performance accordingly. Instinctive Learning contributed to 9.2% of completion percentage3. The second cognitive skill contributing to differences in QB completion percentage was Distraction Control, or the ability of a QB to shield his attention and motor execution in the face of distraction (e.g., hands in the face, hearing footsteps, lineman rolling up on the back of the leg). Distraction Control contributed to 8.5% of completion percentage4. Finally, the third cognitive skill associated with completion percentage in a small but meaningful way was Decision Complexity, or the ability to quickly filter through and execute an action associated with "if-then" rules (e.g., if the corner drops into zone and the backer blitzes, then I dump to the slot, but if the corner covers and the backer drops back in coverage, I hit the RB on a bubble screen). Decision Complexity contributed to 8.2% of completion percentage.
Completion Percentage
Finding the correlation between Impulse Control & interception rate
The next step in the analytical pathway was to run a stepwise linear regression with the dependent variable being interception rate and the independent variables being the 9 isolated cognitive skills in the S2 QB evaluation. Here, the model excluded 8 of the cognitive processes for lack of significance at the p < .05 level. The remaining variable was Impulse Control. Here, the r-value was negative, implying a negative relationship between Impulse Control and interception rate. This totally makes sense… the lower the Impulse Control score, the higher the interception rate. QBs with low Impulse Control scores can't control the impulse to throw into traffic if they feel pressured or make a throw based on only a partial read. They are quick to pull the trigger, try and force throws, and tend not to show much patience in the pocket. The model showed that 24.2% of the variation in interception rate is accounted for by Impulse Control. There are other obviously non-cognitive factors that go into interception rate (e.g., WR runs wrong route, etc.), but a quarter of the interception rate explained by Impulse Control warrants knowing this information before signing a player. Although it did not play an additive factor in the regression model, correlation analyses resulting from the regression showed that Distraction Control also plays a significant role in interception rate, contributing to 11.6% of the variation.
Again, the r-value was negative, suggesting that the lower the Distraction Control score, the higher the interception rate. This too makes sense. When a QB has his attention pulled away by hands in his face, hearing footsteps, or someone grabbing his jersey, he minimizes his focus on his target or doesn't notice defenders in the area, resulting in an interception. Thus, a QB who is wired to be impulsive under pressure and easily distracted has a higher propensity for throwing picks.
Finally, a less than significant factor in interception rate was Tracking Capacity8. When a QB has difficulty with broadening his attention to see across the entire landscape and anticipate where defenders and receivers will move on the field, it becomes hard to predict whether a defender may "intercept" a WR or the ball. Tracking Capacity requires taking limited speed and angle information to make a judgment where two moving targets will intersect. When this is low, QBs make throws where they didn't see a defender or make the correct judgment regarding the speed at which the defender was closing in on the WR.
Interception Rate
Takeaways
There are quite a few take-home messages here about cognition and QB performance. If I'm a front office executive, a scout, or a recruiter, I'm searching for QBs with high tracking capacity, high impulse control, and high distraction control. These 3 cognitive skills alone significantly contribute to a QB's future completion percentage and interception rate. Our next project is to understand certain cutoff scores in the cognitive skills to keep completion percentage above 75% and interception rate below 3% at the NFL level. If you only rely on a QB's completion percentage or interception rate at their previous level of competition, you're analyzing "now" performance and aren't holding anything constant (e.g., level of play/competition). More importantly, doing so is making a context-blind evaluation of their ability to compete and adjust at the elite level. You may have one QB with a 75% completion percentage and less than a 2% interception rate, but if he's playing at a small school in a less populated area or has a WR that can catch anything close to him, those measures don't tell the true story.
You may underestimate the impact of the complexity and speed of the game on these metrics at the next level. Understanding how the athlete is wired for decision-making mitigates the risk of selecting a QB with cognitive capacities that might be exposed at the highest level of competition, where cognitive decisions are pushed to the limit.
For context, and to put together our findings from Part 1 and Part 2 of this article, the 5 NFL starting QBs who have S2 Overall Scores of 90+ with high Tracking Capacity, Impulse Control, and Distraction Control have won 2 Super Bowls, 2 MVP awards, made it to 7 Conference Championships and have an average Career Passer Rating of 100.65… all of these performance accolades, and we have not tested Tom Brady (who isn't included in this analysis).
Results
Tracking Capacity to Completion Percentage [F(1, 25) = 10.037, p = .004; r = .535]
Tracking Capacity & Impulse Control combined to Completion Percentage [F(2, 24) = 12.588, p < .001; r = .716]
Instinctive Learning to Completion percentage [r = .304, p = .061]
Distraction Control to Completion Percentage [r = .292, p = .070]
Decision Complexity to Completion Percentage [r = .286, p = .074]
Impulse Control to Interception Rate [F(1, 25) = 7.973, p = .000; r = -.492]
Distraction Control to Interception Rate [r = -.341, p = .041]
Tracking Capacity to Interception Rate [r = -.280, p = .079]
About Brandon Ally
Brandon has a PhD in neuropsychology and cognitive neuroscience and has published more than 50 peer-reviewed articles and book chapters on brain mechanisms underlying visual attention, perception, and memory. He is the co-founder and Vice President of S2 Cognition, a sports-science company that delivers a leading cognitive evaluation and technology platform to teams and athletes across all major sports at every level of play.