 |
 |
 |
 |
 |
 |
 |
 |
One test wonders: the virtues and vices of single assessment solutions.
Much of my time over the last year has been spent assisting a Fortune 500 retail organization with deploying a variety of staffing assessment solutions. My role includes responding to sales calls from staffing assessment vendors. Given that my client is experiencing double digit growth and receives applications from over 500,000 candidates a year, they receive quite a few of these calls.
In the past 12 months I have listened to or read “sales pitches” from over 30 different assessment vendors. For some people, such a lengthy stream of vendor sales pitches might seem as pleasant as undergoing a series of 30 root canals. However, as a person who has dedicated his professional life to understanding, designing, and deploying staffing systems, I have found this work to be fascinating and informative (okay, I admit to being something of an “assessment nerd”).
I have talked with vendors ranging from publicly held, multi-national staffing assessment companies to small consulting companies with fewer than 10 employees. Much of the knowledge I have gained through these discussions is reflected in the contents of the Rocket-Hire Buyer’s Guide. The purpose of this article is to call attention to the merits and limitations of one specific approach to assessment taken by many of the vendors I have spoken with. This is the tendency of some vendors to advocate using a single assessment tool as the solution to virtually all staffing needs.
Once you get past the glossy marketing materials, the “staffing solutions” offered by many assessment vendors basically come down to a single assessment tool or test. These tests are usually designed to measure a range of personality characteristics, motives, values, and abilities associated with different aspects of job performance. I refer to these broad, multi-use tests as “single assessment” (SA) solutions.
SA vendors usually claim that their one basic assessment tool can predict performance for a very broad range of jobs. They may also offer several kinds of reports for interpreting data collected by their tool to support things such as applicant pre-screening, candidate interviews, and employee development. These reports are often marketed in a way that might lead you to believe that the vendor is offering access to several different assessment solutions. However, when you dig deeper you find that all of the reports are based on data from the same basic tool.
Like most staffing assessment solutions, SA solutions are neither uniformly good or bad. It depends on the unique staffing needs of your organization, as well as the particular nature different SA tools. On the other hand, because they rely on a common “one size fits all” methodology, SA solutions tend to share some common features. The following is a brief summary of issues to consider when investigating the use of SA solutions. Most of the issues reflect both a strength and weakness of the SA approach:
- Off the shelf – By definition, SAs are an “off-the-shelf” solution. The vendor is selling a tool that has been designed to be used with little to no customization for a wide range of jobs. The advantages is that SAs can usually be deployed quickly with relatively low costs compared to more customized solutions. The disadvantage is they are likely to be less efficient and effective than solutions that are configured for a specific job and organization. Much of the assessment content in an SA may focus on candidate characteristics that are less relevant for the particular position you are staffing, thus reducing efficiency.
- Multi-use – One of the nice things about many SAs is that they are designed to be used in a variety of different ways. For example, many SAs provide reports to support both candidate selection and employee development. This is useful for fully leveraging the value of data collected during the staffing process, as well as fostering alignment between staffing and development practices. The disadvantages is that many SAs were primarily designed to support one function (e.g. selection), and are not particularly well suited for other uses. Although they can provide support for other uses, these alternative applications are “secondary” to their primary purpose and it shows.
- One size fits all – Most SAs are designed to predict relatively “typical” job behaviors so that they can be used with as many jobs as possible. As a result, the more unique the job, the less effective a SA is likely to be for predicting its performance. This is perhaps one of the main limitations of SA solutions. Although they may predict performance of a lot of jobs to some degree, they may not predict performance in any one job very well. Many vendors attempt to address this by providing some ability to customize the scoring or reports used by their SA. However there are often serious limitations to these customization efforts since they typically do not involve changing the actual content of the assessment tool itself. Its like trying to tailor an off-the-rack suit – when you cannot change the basic material and design of the suit, there are serious limits to what you can accomplish by changing the hem line.
- Are the old ways the best? – Many of the SAs on the market have been around for years, and have had relatively few substantial revisions to the basic design of their assessment tools. I have seen SA content that has been in use for over 40 years. On one hand, this content has well demonstrated validity – in this sense one might argue “if it ain’t broke, don’t fix it.” On the other hand, there have been substantial advances in the design of staffing assessments over the past 30 years. Just because something is not broken does not mean that is working as well as it might.
- Over interpretation –While SAs can provide valuable information for guiding staffing decisions, there is a risk of managers over-interpreting the test results. This risk is increased by the tendency of some vendors to take an almost “evangelical” stance when talking about the accuracy of their SAs for gaining insight into people’s underlying capabilities. Providing SAs that claim to provide a comprehensive view of all candidates regardless of the job can lead to hiring managers disengaging from the hiring process and instead letting the assessment tool “do the thinking for them”. The result is reduced effectiveness, felt responsibility and accountability for making good hiring decisions.
- Accepting the ideology – Several SAs on the market are based on broad, underlying theories of human behavior. Deploying these SAs may include educating hiring managers on these theories as “the way” to think about employee behavior. I have seen cases where the use of an SA has literally changed how hiring managers think and talk about people. Instead of using adjectives to describe employees, they talk about them in terms of test scores. Although it does feel a bit like “Brave New World”, this is not necessarily a bad thing assuming the theory behind the SA is sound. However, many of the theories that form the foundation of SAs have not been widely tested or accepted by the broader academic community. By using these SAs you may be asking your company to adopt a way of thinking about people that is really more a reflection of one test developer’s beliefs as opposed to well tested principles of human behavior.
- Consulting support – Many SA vendors are relatively small companies. As a result, SAs are often sold using independent resellers in different geographic regions. The advantage of local resellers is that they can often provide high levels of personalized support at a fraction of the consulting costs that are charged by larger assessment companies. The disadvantage is that local resellers are often “test sales people” who lack the expertise and knowledge that consultants from larger assessment companies usually have. In several cases I have found myself talking to SA resellers who know less about how their test actually works than I do.
Few vendors would actually label their products as a SA solution, even though I would argue that being an SA vendor is neither a good or bad thing. When you start looking for SA vendors it does become readily apparent which assessment companies fall into this category. Over 10 vendors included in the Rocket-Hire Buyer’s Guide could be placed into the SA category.
SAs can be effective if you are interested in getting something up and running quickly with relatively little up front costs. They are likely to be particularly attractive to smaller organizations with a wide variety of jobs but relatively low hiring volumes. However use of SAs should be approached with a healthy degree of skepticism. Many SA vendors have a tendency to oversell the value of their tools, and care should be taken to make sure the SA does what it is marketed as doing. Do not assume that just because a SA produces a report listing a certain job title that it actually predicts performance for that kind of job. As with all assessment solutions, ask the vendor for empirical studies demonstrating the validity of their tools. Also ask them to describe the limitations and risks of their SA. When listening to their answer, remember the saying that “something designed to do everything often does nothing well”.