Who Comes To Mind: Balancing Intuition and Availability in Scouting

Who Comes to Mind?

When I say, “great NFL quarterback,” who comes to your mind?

There are plenty of answers to this question, but I’m going to insert Patrick Mahomes as an answer here. It’s an answer that resonates strongly with me when I consider the contents of this article. You could pick a variety of players here and run the same thought experiment.

Mahomes was a divisive player when he entered the NFL. He possessed extremely rare physical traits, with special off-platform throwing ability, arm talent and creativity outside of structure. He also came from a historically maligned Air Raid offense, made plenty of risky decisions that backfired at times and needed room to further develop his natural gifts.

Although he was ultimately selected 10th overall, the public assumption heading into draft night was he could be selected anywhere from the top 10 to the back of the first round.

We know how this story ends. Mahomes spent a year under the wing of veteran Alex Smith, then he took flight all the way to league MVP honors. He is now a Super Bowl champion carrying around a historic contract, and his place as the face of the Kansas City franchise is undoubted.

It’s easy in hindsight to think about why Mahomes should have been given a higher grade by both teams and draftniks. The physical traits, the character and the creative brilliance of his game all point toward success. It’s the same reason why investors can look at a successful company after the fact and say, “Jeez, that would’ve been a great investment!” If foresight is like driving into dense fog, hindsight is like having a car riding your tail and blaring the horn.

People tend to naturally and easily link narratives and causality after an outcome occurs. We can intricately tie the reasons why a player like Mahomes or a company like Google hits a moonshot in terms of return on investment, but we’re inconsistent at predicting who will be next.

As a result, we usually lean on our hindsight as the best indicator of what our approach to foresight should be. It makes sense. If we don’t use our knowledge of the past to predict potential future outcomes, what else is there to use?

This hindsight is so powerful it’s likely to be the biggest brush coloring your future perception of quarterbacks. When you think about great quarterbacks, you’re increasingly likely to think about the wild out of structure plays from Mahomes as you are to think of a calm and cool customer inside the pocket like Peyton Manning.

If you’re relatively new to scouting and don’t easily recall studying a player like Manning, the image of Mahomes for what a “great quarterback looks like” may loom even larger in your perspective of the position.

What is Availability?

In the terms of Nobel Prize winning cognitive psychologist Daniel Kahneman, “What you see is all there is,” meaning that our perception of something is built entirely by what we personally have seen from or know of it.

We don’t know what we don’t know, and we are often content to assume we have the relevant information at hand when facing a new problem. But our individual knowledge is often a puddle in comparison to the ocean of knowledge available outside of our view.

For instance, unless you’ve done the digging to find video and evaluate Joe Namath, the odds are that your lens of “what a great quarterback looks like” will never include Broadway Joe. Until prompted with that last sentence, you may have never considered Namath as worth adding to your model of what a quarterback should look like. What you see is all there is, and until now you probably didn’t see Namath as more than a fuzzy and low-resolution highlight reel with a strong legacy.

This is where an experienced scout with a long line of reports throughout many years can have a significant advantage over an inexperienced scout. The wider the net of quarterbacks studied, the more balanced view an individual can have on what all exists at the position. It’s hard to figure out a puzzle, but it’s easier when you’ve seen similar puzzles in the past.

The problem that is sparked by this availability heuristic (think mental shortcut) is not just failing to seek new information that would improve a decision though. It’s often an issue of latching onto certain prominent features or aspects that are more easily recalled but might not be as good of an indicator of how to handle a situation.

When we face a question, our mind usually easily lobs out an answer seeking something that will suffice, and we can either accept the answer or stop to reconsider.

Think of the question posed at the start of this article—“when I say great NFL quarterback, who comes to your mind?”—an answer comes to mind.

You likely didn’t spend an hour researching and ranking quarterbacks. Your mind was posed a question, it retrieved an answer and if you were particularly careful, you also reconsidered if the player retrieved met the criteria of “great NFL quarterback.” It likely took only a couple of seconds at most to produce this answer.

This is a marvel of associative memory. You didn’t have to think hard, research and strain to answer that question. Your mind filled in the blank very fluently, likely based on the first option that popped into your head.

Try the following exercise. It’s an excellent example of the difference between “fast” intuitive thinking and “slow” deliberate thinking. See how your mind responds to these two equations.

2 + 2 = ?

24 x 17 = ?

With 2 + 2 - poof, it’s there. With 24 x 17, it’s not. 2 + 2 retrieved an answer, and 24 x 17 did not.

One is mental work and the other is not. Or in the terminology of Kahneman—one simply happens to you and the other requires you to take an action to bring forward an answer.

On a question like the great quarterback one, this is not an issue. It’s an easy question with an answer that would likely produce some consensus if you polled a room full of people. Whether the choice is Mahomes, Brady, Manning or Montana, there will be certain players named fairly regularly. It’s one that requires little foresight and pretty much only hindsight. It's not mentally strenuous.

Here’s a much tougher question: “Will any QB selected in the next class develop into an all-pro caliber player?” Pause and consider this question for a moment.

What types of things are you thinking of?

What traits must this quarterback have? Does he need to have a certain style of play? Does he need to come from a certain type of offense or program? What mental characteristics must he have? Does he have to be a certain caliber of athlete or have a certain level of arm strength? Broadly speaking, how does he play football?

This is a much more complicated question than selecting a great quarterback but think about your mental processes when considering that question. You likely have a mental framework of what that type of player will look like.

He isn’t 5’7” or 6’9”. He isn’t 175 pounds or 265 pounds. He may not necessarily need the best arm in the league, but he needs to make certain throws. He has to deliver passes within a certain degree of accuracy to meet your standards.

This mental framework of what an all-pro quarterback should look like takes more strain to draw upon than answering the “name a great quarterback” question. It may be more fuzzy and inexact, but it does exist. And yours is going to look somewhat different from the framework that exists in the mind of other readers based on your own availability of information.

The Error-Balancing Bicycle

The dilemma that emerges with this type of associative ease and general pool of information in our mind is we don’t always consistently and accurately weigh the various aspects involved. In other words—our mental frameworks of “what it should look like” may be miscalibrated.

We unknowingly substitute tough questions for easier ones. We misjudge and misweigh the value of certain characteristics. We have a level of nearly unavoidable “noise” or variability in our judgments from day to day or situation to situation. We carry social and psychological baggage with the decisions and judgments we make. In short, we’re human, and we make mistakes.

I may watch a prospect and see shades of Josh Rosen in his game, which could lead to a far harsher assessment than if I am watching and see shades of Mac Jones. That would be substituting a tough question (how good is this prospect?) with an easier question (who does he remind me of the most and what happened to that player?)

If I missed on Lamar Jackson, Josh Allen or Patrick Mahomes, I may place inordinate weight on the physical tools of a prospect beyond what is reasonable. I may overcorrect for my prior misses. That would be misjudging and misweighing certain characteristics.

I may see that all of my peers grade Spencer Rattler as the favorite to go 1st overall heading into his sophomore year, and I may feel some social pressure to conform. I might temper my grade to avoid being a social outlier or receiving criticism for my opinion. That's social and psychological baggage.

These are the types of problems that come up when trying to effectively evaluate a player without swaying too far to one side or the other. Lean too far towards playmaking quarterbacks with rare traits and you may not give enough credit to consistency, processing and ability within structure.

Lean too far towards an immobile pocket passer with modest tools and you may get a quarterback who can’t elevate his teammates or win when things go awry.

To use the analogy of University of Pennsylvania professor Phil Tetlock, an expert on the topic of forecasting and improving judgment, this type of decision making is a lot like riding a bicycle. Tip too far in either direction and you end up on the pavement. It takes practice, and it takes scraping your knees a lot.

The reason I picked Mahomes to begin this article is that he was one of the first quarterbacks I ever evaluated. I graded him as a seventh-rounder. There’s falling off the bicycle, and then there’s driving it into oncoming traffic. I got run over, and rightfully so.

I wanted a pretty-looking pocket passer, who emulated Tom Brady to a tee. The question I answered wasn’t “How good will this quarterback be.” The question I answered was “How much does this quarterback look like what I think a quarterback should look like?”

That mental framework of “what a quarterback should look like'' was so completely broken in my head that it led to a terrible evaluation. The information available to me at the time was crap, and I didn’t see it.

Combatting Availability Limitations

It’s humbling to realize how much information is outside of your view. It may be frustrating as well, given how useful that information would be as part of your mental framework for evaluating players. There are some good options for dealing with this problem though.

Using base rates

A base rate is relatively easy to express, and it provides a good starting point for any projection. It’s what happened in the past expressed as a percentage of the total population. Here is an example.

There were 22 running backs selected in the 2022 NFL Draft. Of those 22 running backs, 20 of them ran a 40-yard dash of 4.65 or faster (hand time). The two exceptions were Notre Dame’s Kyren Williams and UCLA’s Brittain Brown.

The base rate would be 20/22 (91%) for running backs drafted having a 4.65 or better in the 40-yard dash.

Interestingly enough, the base rate of 20/22 is the same for drafted running backs weighing 200 or more pounds (pro day weights). The two exceptions, who weighed less than 200 pounds, were Williams and Missouri’s Tyler Badie.

The power of the base rate is getting a larger and more representative view rather than looking inward at just one prospect to make a judgment. It’s using a large net of info with a clear historical value rather than just projecting from within your mental framework.

Regardless of how much you love a running back’s tape, a draftable projection should be very tentative if they weigh less than 200 pounds and run slower than a 4.65 40-yard dash. There are rare exceptions (like Williams), but the vast majority of players who fall below those two thresholds will fail.

For reference, there were 19 running backs who weighed less than 200 pounds and ran slower than 4.65 in last year’s class. Williams was drafted in the fifth round. The 18 other running backs did not even receive a tryout.

On the flip side, there were 53 running backs who weighed more than 200 pounds and ran faster than 4.65 in last year’s class. 18 were drafted, 20 received a deal as an undrafted free agent, seven received tryout opportunities and only eight received no NFL opportunities.

The base rate of a running back being rejected at less than 200 pounds and slower than 4.65 is 95%.

The base rate of running back being rejected at 200 pounds or more and faster than 4.65 is 15%.

The NFL wanting big and fast people may seem like a common sense thing, sure, but it’s easier to forget about common sense when a player does a few things you really love on the field.

A few tough runs or a splashy explosive play can commandeer your thinking, especially if you lack experience and are working from a smaller base of knowledge. I did this constantly throughout my first few years evaluating players, and it led to horrible results. Base rates are like pouring cold water on the “inside view.”

Seeking perspectives and learning from others

Another great option to help generate a more complete view is actively (and constantly) seeking the experiences and perspectives of others, then incorporating them into your own views when applicable.

Asking good questions of knowledgeable individuals will help to fill out your own mental framework at an accelerated rate, but even those with less experience and knowledge on a subject can be very useful to seek insights from. They can generate questions or responses that push at basic assumptions, and getting a chance to poke and prod on both general and specific topics can improve your thinking.

Here is a question that a non-expert could generate that could still be useful to think about and consider. Although, an expert would likely have an intuitive and immediate response.

“Why is the NFL Combine important at all? Is it worth caring so much about?”

An expert could list a dozen reasons for why the Combine is important and why it takes place, but it’s harder for them to rethink such a basic assumption of the Combine’s usefulness.

This isn’t to say those reasons are not valid. It’s to ask “Am I overvaluing the Combine? What would cause me to overvalue it?” rather than assuming the Combine is valued properly and letting that assumption fade into the background.

The COVID-19 pandemic forced NFL teams to adjust away from their typical practices and begin interviewing prospects over Zoom. While some teams still prefer to do traditional interview practices, many have started taking advantage of the new opportunities with Zoom.

While it ultimately took a pandemic to create this change, the fundamental question of “Why do we do all these interviews in person?” was something that could have been poked and prodded prior to a forced change.

Expertise and experience generate many answers, but sometimes those answers conflict. One experienced scout may say they regret grading an undersized defensive tackle too high, and another scout may say they regret grading an undersized defensive tackle too low.

There’s no “right” answer here, just two separate experiences that can help us ask “What makes an undersized defensive tackle succeed or fail?”

A large part of this begins with an “actively open-minded” approach to thinking about problems, as many in the field of cognitive psychology call it. For instance, if you enter a conversation about quarterback play believing there is only one way to play the position, you won’t learn and incorporate a different view from your own prior beliefs.

The first step to learning something new is believing you may not already know the answer. If you cannot do that, then you will be unable to use the valuable and diverse insights of others.

Deliberate practice and accepting feedback gracefully

Last, is simply taking the time and effort to practice and being capable of taking lumps throughout the process. Going back to the example of the experienced scout with a long line of reports, there is significant value in just doing something for a long time, working hard to assess the feedback from it and then using that to improve your mental frameworks.

But it’s not as simple as just chopping away for ages and hoping time is the answer to eliminating failures. To quote the article on deliberate practice that is attached at the bottom of this article, “It may appear that excellence is simply the result of practicing daily for years or even decades. However, living in a cave does not make you a geologist.”

To deliberately practice is to identify areas in which improvement is needed, to be systematic about working to improve in those areas, and to recognize the feedback (positive or negative) and embrace it without excuses. This does generate some issues though.

Scouting is often a slow feedback process, which makes it hard to know how well you are doing. If you grade a player as a starter, it may take three or four years to know whether or not that assessment was correct. Compared to something like golfing, where you know very quickly if your swings are slicing balls way off course, it is harder to ascertain whether an assessment is a hit or a miss.

This slow feedback process also makes it much harder to decipher cause and effect. Send a ball into the water as a golfer? You can pick apart what happened in your swing that caused such a poor outcome. You evaluate a player, and then reassess in three years? There are a lot more variables, there is a lot more out of your control, and it’s harder to determine cause and effect.

Was it an injury that caused the player to fail? Was there something off the field I didn’t know about? Did I misjudge their physical attributes? Would it have been different if they were with another team?

Furthermore, accepting feedback is hard. If it is positive, it’s easy to get caught up in a victory lap without assessing the ways in which you could have (or even should have) been wrong. If it is negative, it’s difficult to accept that you were wrong, that you made a mistake, and that you are less impressive with your judgments than you would like to be. It’s a blow to your identity as an intelligent and careful decision-maker, but it shouldn’t be.

You’re looking through the microscope of your own perspective, and it takes time to broaden that view into something more all-encompassing. The only way you can fail is by saying, “No thanks, I think I see it well enough already.”

Supplementary Materials

Daniel Kahneman on “What You See Is All There Is” (audio)

A more detailed explanation of the Availability Heuristic (text)

Farnam Street on base rate insensitivity (text)

Annie Duke on base rates in betting (audio)

More on deliberate practice and creating expertise (text)

Twitter responses to the “Who Comes to Mind?” question (text)