« Back to the December 2004 Explorer newsletter

FEATURE

Business Intelligence – The Future of Hiring Metrics

By Charles Handler, Ph.D.

Technology has changed almost everything about the way we do business. It's hard to argue with the fact that advances in both hardware and software have created fundamental changes in the way businesses are run. Technology now plays an indispensable role in helping businesses to accomplish their strategic objectives, and thus generate revenue.

One of the most interesting ways that businesses have been able to leverage technology to increase profitability is in the application of a set of tools and techniques known broadly as "business intelligence" (BI for short). While business intelligence has been in use for a long time in some industries, increases in software sophistication and the success stories of those who have benefited from it have led an increasing number of organizations to begin applying business intelligence to help them increase their profitability.

So what exactly is business intelligence? Whatis.com offers the following definition:

Business intelligence (BI) is a broad category of applications and technologies for gathering, storing, analyzing, and providing access to data to help enterprise users make better business decisions. BI applications include the activities of decision support systems, query and reporting, online analytical processing (OLAP), statistical analysis, forecasting, and data mining."

Business intelligence applications can be:

• Mission-critical and integral to an enterprise's operations or occasional to meet a special requirement
• Enterprise-wide, or local to one division, department, or project
• Centrally initiated or driven by user demand

The term was used as early as September 1996, when a Gartner Group report said:

By 2000, Information Democracy will emerge in forward-thinking enterprises. Making sound business decisions based on accurate and current information takes more than intuition. Data analysis, reporting, and query tools can help business users wade through a sea of data to synthesize valuable information from it — today these tools collectively fall into a category called 'Business Intelligence.'

Webopedia.com offers the following definition:

The term Business Intelligence (BI) represents the tools and systems that play a key role in the strategic planning process of the corporation. These systems allow a company to gather, store, access, and analyze corporate data to aid in decision-making. Generally these systems will illustrate business intelligence in the areas of customer profiling, customer support, market research, market segmentation, product profitability, statistical analysis, and inventory and distribution analysis to name a few.

The above definitions are relatively broad and can apply to any number of organizational processes or systems. But no matter what the application, BI relies heavily on an important byproduct of the use of business technology systems: data. While data was once looked at as having limited value, increased sophistication of technology has allowed data to become the fuel that can be used to feed sophisticated analytical processes. The end result is that data that may have once been discarded or left unexamined has been given new life, because it can actually help organizations to systematically make more intelligent business decisions.

At the present time, BI is being used mostly for solving problems involving distribution, logistics, sales, and marketing. While BI (as defined above) has yet to be applied to hiring, I feel that it will eventually help offer businesses a competitive advantage. After all, hiring is really a decision-making process, and there is a significant amount of data that is generated during this process. It seems only natural that this data should be leveraged to help improve the quality and accuracy of decisions that result from the steps in the hiring process.

One of the reasons I think that BI is a natural fit for hiring is that we I/O psychologist types have been using a form of BI for decades to help organizations better understand the relationship between the elements of the hiring process used to support decision making (usually tests or assessments) and the outcomes that have value for the organization (usually turnover, or increased job performance). We refer to this process as "test validation."

(A full discussion of test validation is beyond the scope of this article. If you want more detailed information on this topic, my colleague Dr. Wendell Williams has written many great pieces on the topic of test validation for ERE. I also recommend that anyone interested in learning more about this topic take a look at the U.S. government's Employer's Guide to Best Practices for Testing and Assessment.)

Test validation is useful because, in a nutshell, hiring is all about accurately predicting which applicants will be the best fit for the responsibilities associated with a specific job. Using valid tests to help make these predictions is advantageous, because the more valid the test, the more accurate the predictions made based on the test will be.

So what makes a test valid? A test is said to be valid if it measures what it is supposed to measure. In the case of hiring, "what it is supposed to measure" is the candidate's ability to perform the job he or she is applying for. Validating a test, then, is the process of demonstrating that the test is an accurate predictor of job performance. Assuming the test is job related, the better the applicant scores on the test, the more capable they will be of performing the job they are applying for.

Now for the business intelligence part of this discussion. While not all forms of test validation are based on data, for decades we I/O psychologists have been using data-based test validation to help organizations understand the level of accuracy in hiring decisions provided by the use of tests or assessments. This is done via a statistical study known as a "criterion-related validation." In such a study, the statistical relationship between test scores (the predictor) and job performance or other job-related outcomes such as turnover (the criterion) is examined. The data analyzed in this process provides a good idea of how accurately the test is in predicting performance.

There are two major reasons why data demonstrating this relationship is of critical important for businesses:

1. Legal defensibility. A properly conducted criterion-related validation study provides statistical proof that a test is measuring what it is supposed to. The foundation of legal defensibility in hiring is job relatedness, so documentation in the form of statistical proof is a good idea.
2. Understanding ROI. This is where the BI part comes in. Data-based test validation allows organizations to gather statistical evidence that will help reveal the increases in ROI that are associated with the use of a specific test or assessment. This means that the data collected can allow organizations to understand the monetary outcomes associated with increases in predictive accuracy provided by the use of a test or assessment. The ability for organizations to better understand the outcomes associated with increased accuracy has been a selling point for the use of assessments for quite some time now. In fact, decades worth of test validation studies have continually reinforced the conclusion that, when used properly, pre-employment testing can provide serious ROI.

While there are many benefits to test validation, and it is a best practice I fully endorse, when it comes to hiring I feel that the concept of BI is much, much bigger than simple test validation. Hiring an employee is the result of a process not just one test or even a series of tests. This means there are all kinds of opportunities to use data to evaluate the accuracy of the process as well as the outcomes made via the process.

Think about it, hiring an employee involves the following:

• Sourcing/Recruiting
• Screening
• Assessments
• Interviewing
• Final decision making
• On-boarding

Each of these components of the hiring process has value individually, but they also function together to help organizations make the hiring decisions needed to help fill open positions. Hiring decisions are serious business, as they provide the human capital required for organizations to execute their strategic objectives. So each hiring decision has a direct link to outcomes both objective and subjective that are valued by the organization. This means that the more an organization understands the relationship between each and every element in it entire hiring process, and the outcomes needed for success, the better equipped it will be to build a process that systematically provides value.

When it comes to BI and hiring, raw material is not a problem. There are mountains of data that are associated with each step of the hiring process. The real value in BI comes in the collection and analysis of all of this data and in the exploration of linkages between these data and all manner of post hire outcomes.

Applying BI to hiring in the same manner as it has been leveraged for other functions within organizations will require hiring personnel to think outside of the box. In its truest form, BI goes beyond the tactics that have been utilized to this point in the evaluation of the hiring process — far past the use of simple metrics such as cost per hire, or even more complex ones such as test validation.

BI is really about data and the ability to look into it in order to find valuable trends and relationships. In a way, BI is like a fishing expedition, because it often requires a bit of experimentation and casting about to land something meaningful. As with fishing though, the value of catching a "big one" can easily offset all the preliminary effort that goes into it.

Please understand that I am not advocating that organizations begin setting off on any wild data-crunching goose chases. At the present time, there are very few companies that provide the software needed to apply BI to the hiring process. We still have a lot to learn about how BI can be used in a practical manner to help extract value from the hiring process. Mark my words, sometime in the not too distant future there will be vendors offering technology-based BI solutions for helping organizations optimize their hiring process and more fully understand the ramifications of their hiring decisions.

Despite the immaturity of the use of BI for hiring, there are some things we hiring professionals can learn from this cool new concept. For now, here are some simple steps you can follow to begin looking beyond the simple evaluation individual parts of the accuracy of tests towards an evaluation of the accuracy of your entire hiring process.

1. Define outcomes. Outcomes can vary significantly. While job performance may be one outcome, there are others. Without understanding valued outcomes, it becomes difficult to attach value to data you collect.
2. Evaluate each step of your process relative to these outcomes. This means looking critically at the relationships between each step in your hiring process and the important outcomes associated with it. Each of the following parts of the hiring process provides opportunities to collect data:
   • Recruitment branding/advertising
   • Qualification screening
   • Resume evaluation
   • Assessments/testing
   • Interviews
   • Overall decision making
3. Look for meaningful relationships in this data. Why not begin looking at the impact that each of the above steps is having on the others, as well as their impact on the ability of your hiring process to provide outcomes valued by your organization?
4. Make adjustments based on what you find. The spirit of Six Sigma suggests that organizations listen to the data they uncover and use it to continually improve the quality of their processes.
5. Provide data with a life after the hiring decision has been made. It is important to understand that the data collected during the hiring process should not be discarded once hiring decisions have been made. This information has value both individually (for the ongoing development of the employee) and collectively (for looking at trends in the relationships between hiring data and outcomes).

In conclusion, BI is a mindset that involves the understanding that all of the information collected during the hiring process has meaning. Once you have adopted this mindset, you will find many opportunities to uncover valuable relationships. If your organization does not have any experience with this kind of thing, test validation is a great place to start. From there the possibilities are almost endless.