« Back to the October/November 2004 Explorer newsletter

Best Practices

Whether it’s a Case Study or a Step-by-Step Process, we like to publish material that focuses on how to get things done the right way.

Selection Strategies: The Pros and Cons of a Top-Down Approach

By Tommie Mobbs, Ph.D.

In the field of employee selection, specialists have long used the guideline that “more is better” and “those with the most”, i.e.; those who score highest on tests, should be the best candidates and first to be hired. This would be called a top-down selection strategy. For instance, when selecting for a position requiring strong cognitive abilities, it seems reasonable to predict that the “smartest” applicant would perform better than the next smartest, and so on. In other words, a linear relationship between selection test scores and job performance is assumed. However, there are some problems with this assumption.

Example Problem #1

Every predictor is not related to performance criteria in a linear fashion. Take for example personality traits, which have enjoyed a resurgence of interest for predicting future job performance. One study of a major sales and service call center showed significant performance and turnover problems when assuming a linear relationship between personality traits and performance. Here’s what happened:

For approximately nine months, the call center used a personality-based assessment to find candidates for sales positions, but they were surprised when the results didn’t measure up to expectations. The then-frustrated HR Director explained, “I had been told by the (assessment) vendor that, all else being equal, candidates who scored higher on the “Extraverted” scale would make better salespeople.”

Intuitively, this advice seemed to make perfect sense. Individuals who are energetic and talkative should have greater success in sales, right? However, the HR Director’s first-hand experience led her to another conclusion: candidates who scored in the mid-range in “Extraversion” outperformed both low and high scoring candidates. What was the reason for this apparent anomaly? After qualitative analyses, the Director found that candidates assessed as extreme “Extroverts” grew quickly bored with the repetitive assignment of targeting a customer base of small business owners who had already been solicited repeatedly. “Those high on the extraversion scale wanted more exposure to change,” the HR Director noted, “resulting in unexpected lower sales and higher turnover in the group of highly extraverted individuals hired.”

Example Problem #2

Here’s another recent case study (Manley & Mobbs 2004)¹, examining Air Traffic Controller performance among FAA Academy trainees. This study measured trainees using a biographical questionnaire tapping the personality trait of “Conscientiousness”, which included facets of ‘work orientation’, ‘dutifulness’, and ‘preference for meaningful order’. In the study, three out of four “Conscientiousness” subscales had relationships with air traffic training performance that proved to be non-linear! In this sample, those very low or high in “conscientiousness” outperformed moderately conscientious trainees.

By using a top-down strategy here, some of the most qualified candidates will be overlooked. The FAA has been advised about the potential pitfalls of hiring with a top-down strategy when targeting personality traits.

The Solutions

A seemingly infinite number of variables are associated with human thought, decision-making, and behavior – which make the use of behavioral testing anything but straightforward. In the examples provided, Presidio Sciences was able to provide objective insights, as well as solid science, to help both the call center and the FAA think about their current top-down hiring assumptions and strategies.

Our cited cases do not demonstrate that all off-the-shelf assessments scored linearly are necessarily bad. However, caution is advised in making sure these assessments are working in your organization and culture.

A procedure to examine how well an assessment might fit into your selection process is termed “Test Generalizability”, or “Validity Generalization”. This examination of an assessment is often conducted by Industrial/Organizational Psychologists, with a primary goal of determining whether or not a particular test was developed in a way that makes it relevant in another organizational setting.

Another approach is to score selection tests in such a way that they do not assume this linear relationship between predictors and performance criteria. Businesses interested in maximizing performance predictions within their hiring process through scientific validation are best served by Industrial/Organizational experts who design selection systems that incorporate validity generalization and non-linear prediction.

About the Author

Tommie Mobbs, Ph.D. is President of Presidio Sciences, an I/O consulting firm grounded in ‘good science’, helping organizations large and small to find, hire and retain the optimal workforce for them.

1. Manley, G. G. & Mobbs, T.A. (2004). Investigating the linear predictor - criterion assumption of biodata scaling. Southern Management Association 2004 Annual Conference Proceedings.