« Back to the October/November 2004 Explorer newsletter

FEATURE

Overview of the Technology Based Assessment Marketplace

By Charles Handler, Ph.D.

My efforts to stay on top of the online screening and assessment marketplace have proven to be an interesting ride. This is a very challenging market that is not easy to sum up in a few simple words. The factors that keep the picture a bit on the cloudy side include:

1. Broad scope and fragmentation- Screening and assessment involves a wide variety of different products that are offered by many different types of vendors.
2. Proliferation of smaller vendors- Assessment has its roots in small boutique test development firms so there are a large number of home grown assessments currently in use. These can be hard to keep track of but collectively do account for a significant amount of revenue.
3. Lack of education- There is no shortage of junk floating around this marketplace. Unfortunately, it often takes a trained eye to separate the good stuff from the bunk.
4. Continual refinement- Many firms are continually working hard to address their weaknesses. This means that vendors may significantly upgrade the amount and quality of their offerings from one year to the next.
5. Increased attention from ATS market- The glaring need for ATS companies to provide tools that can help increase quality while demonstrating an impact on the bottom line has been a major factor in the emerging development of screening and assessment. While this is a positive thing, many different strategies have been used often making it difficult to predict movement in this area.

Despite all these pitfalls, I am happy to see the online screening and assessment market beginning to mature as demand for these tools increases. Interestingly enough, some of the very issues that make this a challenging market to keep up with are also responsible for increased demand for screening and assessment tools. For instance, the single biggest driver of recent demand for assessment tools is the fact that ATS customers have been left high and dry when it comes to the ability to demonstrate ROI from their investment in this technology. Increased awareness of the existence of a solution to this problem has come from both vendor marketing efforts and the efforts of a small but dedicated group of subject matter experts who are working hard to provide educational opportunities designed to help demystify assessment.

In an effort to get my arms around the impact of the forces that are creating change and movement in the online screening and assessment market, I have created a simple model to help track the current landscape. This model is based on 2 axes that are crossed to create 4 distinct quadrants.

These axes include:

Axis 1: Technology level (Vertical Axis)- This axis represents the sophistication of the technology that underlies a vendor’s products. The continuum here runs from low technology defined as either paper and pencil or rudimentary web based technology on one end to more complex asp tools that offer sophisticated functionality at the other. More technologically advanced systems tend to provide the ability to deploy content in a highly flexible, modular fashion while also supporting the ability to do more with the data that is gathered during the assessment process.

Axis 2: Quality of product (Horizontal Axis)- This axis is more difficult to define in a precise manner. For the most part it describes the vendor’s approach to creating their product line. Higher quality products are based on solid theories of predicting job performance, were developed using a strict process that follows a set of best practice guidelines, and have been validated to demonstrate their ability to predict job performance. Lower quality products tend to range from products that are effective but have been created as “one size fits all” solutions all the way to products that are complete junk because they lack any of the characteristics that define a viable assessment tool.

Crossing these axes creates 4 quadrants. These include:

Quadrant 1: Low Tech/Low Quality- This quadrant is one that, for the most part, is to be avoided at all costs. There little good going on here. There may be some vendors in the top right of this quadrant who are can provide value assuming that the consumer does not need advanced technology.

• Who is likely to be in this quadrant?- The same characters that have been bilking people for years. Most of these folks have invested in enough technology to allow them to use the web in a rudimentary fashion. Most of the action in terms of low quality testing is actually occurring in Quadrant 4 where unscrupulous firms are looking to cash in on the increased popularity of technology based assessment tools. There may be some marginal products in the top right corner of this quadrant that can add value if used in the right situation, but these usually represent products that cut corners in an attempt to make using assessment “easy” and “fast”.

• What will their products look like?- This quadrant is populated by solutions that include techniques such as handwriting analysis, color analysis, Myers Briggs type indicators or other legitimate assessment instruments that are not suitable for employee selection purposes, and manner of pseudo scientific tests. About the best you can hope for here is a homemade personality test that may be marginally predictive of some basic job related behaviors.

• Whose needs will they fit?- No one’s. Most likely to be adopted by small to medium sized companies who don’t know any better.

• How is this quadrant evolving?- Low quality assessments are nothing new. They have been making things hard for vendors of good assessments for a long time. This quadrant will be relatively stable or will actually decline as more and more vendors look to invest in moving faddish assessments online. While some vendors in this space will make an investment in technology, most will not have the capital to create technology systems that are anywhere close to the market standard and thus will stay in this quadrant.

Quadrant 2: Low tech/High Quality - There are a good many firms who are currently in this quadrant. This is an OK quadrant to be in because there are many opportunities for organizations who offer high quality products.

• Who is likely to be in this quadrant?- This quadrant is made up two types of organizations, firms who have been making a living creating customized assessment tests for decades and test publishing firms. There is a wide range in terms of size within both of these categories. For instance, customized test development is a staple for firms of all sizes from small shops all the way up to large world wide consulting firms. Test publishers range from smaller firms with only a few tests all the way to global publishing giants.

• What will their products look like?- This quadrant is defined by a very test-centric mentality. That is vendors in this quadrant are focused squarely on the creation and validation of individual tests to be used for selecting employees. Most firms in this quadrant do offer some web based capabilities but they are not nearly as sophisticated as those of the firms in quadrant 3. Vendors in this quadrant are likely to follow established best practices for test development and validation. Test publishers are likely to have a broad catalogue of assessment tests that have very solid validation history and will work very well when used correctly.

• Whose needs do they fit?- Firms in this quadrant is able to serve a wide variety of needs. Much of the activity here is carried out by firms who fulfilling small to medium volume testing needs of all types. These firms have the resources and experience to develop highly customized tests and also have a decent inventory of tests that can be used off the shelf with a slight bit of modification. The high quality of content provided by firms in this quadrant is also useful for larger scale testing initiatives. Firms in this quadrant lack the bandwidth to support this type of scale. For this reason many vendors in this quadrant end up partnering for companies who are interested in creating high tech assessment platforms but lack the resources to develop the content to be delivered by the test system.

• How is this quadrant evolving?- Vendors here are moving towards developing more refined and advanced technology capabilities due to increasing demand for technology based tools. The better capitalized firms in this area are making great strides in this regard. This is especially true of test publishers. A few years ago most major test publishing houses had very little to offer in the way of technology. Now some of the larger firms have developed very sophisticated technology platforms. However, most test publishers have a very test centric mindset and thus their products are still not as advanced as some of the vendors who populate quadrant 3. This will remain an active area as there will always be a demand for custom made tests as well tests that can be purchased in a transactional manner. Still, this quadrant will continue to be marked by vendors who are seeking to upgrade their technology level and move into quadrant 3 where much of the action is currently taking place.

Quadrant 3: High tech/High Quality- This is the quadrant that represents the future of online assessment. Vendors in this quadrant understand that the value in assessment comes in creating a system which takes a holistic view such that the sum capabilities of the system are greater then the individual contribution of technology and content individually.

• Who is likely to be in this quadrant?- There are presently very few firms who are at the far end of both axes in this quadrant (top right corner). Firms who populate this corner of the world for the most part were founded as technology organizations who offer a testing product as opposed to organizations who started out in the testing business. Many of the vendors in this space got started by purchasing content created by vendors in quadrant 2 and using it to build out their product. Others have hired the resources to create their own test content. At the lower left of this quadrant reside companies of various types who are pushing to move up from the top right of quadrant 2 based on a realization that a blend of technology and content presents opportunities to demonstrate significant ROI to customers.

• What will their products look like?- There are a wide variety of products represented in this quadrant. The most advanced of these have left behind the test centric mentality for a viewpoint that is driven by the need to provide a flexible modular approach to assessing competencies using a variety of different techniques. This type of assessment is most often coupled with a technology platform that provides candidate management functionality as well as advanced reporting capabilities. The firms in this quadrant can extend the footprint of assessment towards the initial point of applicant contact (website or job board) as well as towards the management of applicant data via integration with an ATS/candidate management system. Firms in this quadrant also use technology to help their clients clearly understand the ROI associated with their staffing process. Finally, this quadrant also houses firms with advanced products that are based on quality content. These include high end web based simulations and other innovative tools that are creating new paradigms for assessment.

• Whose needs do they fit?- Vendors in this quadrant can serve companies of all sizes. They typically have the bandwidth to deliver to larger clients but many offer simpler systems that can be scaled down to serve smaller clients as well. There is an increasing trend towards productization in this quadrant with firms trying to take advantage of demand that is arising in specific verticals.

• How is this quadrant evolving?- These are the folks that are really advancing the practice of selecting individuals for jobs. They are thinking 5 years ahead and are funding this effort via the creation of tools that use technology to collect data that will help organizations to understand how to find and document the value associated with their staffing processes. These firms understand that the next stage of evolution in technology based assessment will be in expanding the value provided by staffing related assessments via integration with other key HR functions such as training and performance management.

Quadrant 4: High tech/Low quality- For the most part this quadrant is a bust if you are looking for top notch stuff. However, the firms in the top right of this quadrant may actually have products that are somewhat useful but are probably far from the optimal solution.

• Who is likely to be in this quadrant?- This quadrant is an interesting one because it contains firms who actually have viable products as well as those who are offering nothing but junk. To the left side of this quadrant are the folks you really need to look out for. They are likely to spend lots of money creating and marketing their high tech systems but these systems will be essentially useless because they aren’t loaded with usable content. Fortunately there are few of these companies as it is simply too costly for an organization to move to and remain in this quadrant without a viable product. Most of the vendors in this quadrant are in the top right. These firms have a good technology system and a viable product that is not total junk but may cut corners in some way. Most of these products can provide some value, however often times they do not represent the optimal solution.

• What will their products look like?- Products in this quadrant range from the same junk found in quadrant one boosted but with a higher technology level to actual viable products. On the good side of this quadrant are companies that use technology to help make the process of selecting and implementing assessment “fast and easy” By taking this tact, they are sacrificing some predictive accuracy in exchange for the ability to use assessment with less up front work.

• Whose needs do they fit?- Products in this quadrant range from doing no one any good to helping companies who may need a fast easy and scaleable process for helping provide some rough guidelines during the hiring process. This is a difficult quadrant to shop in due to the combination of the attractiveness of the vendor’s technology based value proposition and the possibility of buying a substandard but sexy looking product.

• How is this quadrant evolving?- This is a quadrant that will be growing just as fast as quadrant 3. As the need for cool, innovative, high tech solutions begins to drive the marketplace there will be an increasing number of firms attempting to cash in. While some vendors may actually offer products that are viable in some situations, others will continue to offer empty promises.

At the end of the day I think the movement in this market will be away from quadrants one and two (low tech) and towards quadrants three and four (high tech). The content side of things will remain an interesting area to watch. At the present time most of the content that is being delivered as part of online assessment is either stuff that has been around for years and has been loaded into a technology based delivery system, or is content that has been created using the same set of guidelines that test developers have been using for years. Neither of these is a bad thing at all but I am really interested in movement towards the development of new types of content that are completely new and innovative. Once these begin to appear the real challenge will be to determine if they belong in the wastelands of quadrant 3 or if they are worthy of a position in quadrant 4.