« Back to June 2004 Explorer

New Product/Service Spotlight

At Rocket-Hire our research efforts offer us a unique opportunity to preview new ideas and products. Please contact us at info@rocket-hire.com or call (504) 236-7259 to learn more about the ability to use this product to help you demonstrate the impact of strategic staffing assessments to your bottom line.

NEW - Talent Curve Simulator

Introduced byTom Janz, Ph.D., Chief Science Officer
(817) 675-1553
TomJ@BDT.net
www.BDT.net

Beyond Staffing ROI:Talent Curve Analysis and the Bottom Line

We have known how to measure basic mental and physical abilities since the 20s with only minor refinements since the 50s. Most of what we know about how to measure personality has been with us since the 60s, as has our knowledge of how to design and score work simulations. Since the seventies, not much that would improve how we conduct in-person behavioral interviews has emerged. Yet, in a recent survey, normed psychometric assessments guide less than 5% of all hires made by large private corporations. While many organizations have adopted patterned, behavior-based interviews on the surface, interviewing practice in the field remains highly variable, with some hirers using interview guides—and many not, and with some being trained—and many not, and with some probing to reach behavioral detail and others not. Why?

Surely the logic of “measure twice, cut once” ought to be enough to compel employers to use selection practices that actually work. Most companies trumpet the importance of their human resources in annual reports, and then allocate whisper budgets to hiring and developing those resources. Hiring a technical/professional workforce of 100 field service reps at an average salary of $70,000 over an average tenure of 6 years impacts a spending stream of $42 million. Yet, the practice of picking this workforce often gets left to unprofessional personality profiles, vague endorsements by questionable references, and gut-check interviews.

Selection science itself has been partly to blame. While the production manager argued for new equipment based on increased production at a reduced unit cost and the marketing manager argued for branding budget dollars based on increased sales, the HR manager was left arguing for better staffing based on the value of hiring good people, with no dollar benefit to cite.

Back in the fifties, Brogen developed the “dollar criterion” and in the sixties, Cronbach and Gleser produced the utility equation—the math that computes the annual dollar value per hire returned by a selection method, given the selection accuracy, candidates per opening, and the variance in the dollar value of job performance. Even then, financial discussion of the value of selection methods remained academic until the late Seventies and Eighties, when Frank Schmidt and John Hunter refined and popularized practical methods for estimating selection accuracy and the variance in the dollar value of performance, with help from Casio, Boudreau, and Alexander.

Since the mid 80s, HR leadership could calculate the dollar benefit arising from an investment in pre-employment testing or an interview training program, to go along with the spending request, but did so only rarely. Curiously enough, the public sector, always under the gun to justify its budgets, cited staffing program utility more often than the private sector. Training in IO Psychology has included staffing utility analysis since the 90s, but training as an HR professional does not take graduates far enough into mastery of the topic to face a boardroom full of snarling “C” level executives anxious to gain budget approval for their own pet programs.

Boudreau and Ramstad (2001) noted that the increase in academic and professional articles focused on staffing utility analysis seen in the 80s, dropped off during the 90s. In 1994 Latham and Whyte reported that that utility analysis actually reduced managers’ reported support for a hypothetical selection program. In their study, they prepared a description of a hypothetical selection program, and had 143 experienced managers rate the likelihood they would adopt a recommended selection program when that recommendation did or did not include a financial utility analysis. In a follow-up study reported in 1997, Latham and Whyte varied the source of the expertise behind the staffing utility analysis—from a “trusted advisor” to written comments by a psychologist to a video-taped presentation by a psychologist with an opportunity to ask questions. Once again, contrary to what one might hope, the staffing utility expert produced the least impact on the experienced manager’s acceptance of the recommendation to adopt a valid selection program.

This author was invited to join the Latham and Whyte 1997 study, but declined when Latham insisted on presenting the utility findings as academics usually do. He specified presenting a large final utility dollar result representing the projected dollar utility delivered by the proposed selection program, with a page full of formulae and derivations to back it up. It should surprise no one that having an expert from afar announce that the financial benefit resulting from hiring sales staff of 200 based on sales test is $28,000,000 for a total cost of $ 95,000. When asked, “How did you come up with the $28M”? the answer references the utility formula. “We multiplied the number of openings times the hiring gain (based on the number of candidates per opening) times the average tenure times the hiring accuracy (validity correlation) times the standard deviation of annual performance in dollars, and then subtracted the costs of the test.” It’s just too confusing coming all at once. What is hiring gain? What is hiring accuracy? What the is the standard deviation of annual performance in dollars? While there are precise answers for each of these questions, they will float way over the heads of the typical operations executive, never mind HR VP. So what to do?

If you can’t calculate, simulate. Monte Carlo simulation offers the opportunity to observe complex systems as work as statistical models, allowing the engineer to test the impact of wing shape on hypersonic drag or the economist to observe the impact of a 10% import tax on the production of domestic vehicles, without having to actually fly the plan or impose the tax to see the result. This paper introduces two Monte Carlo simulators designed to bridge the gap between the characteristics of a hiring system we can measure and the financial outcome that results from the improved job performance.

The Personal Value Simulatortables the annual performance value for 100 candidates, showing which of the 100 candidates would be hired, a pair at a time, using each of two selection methods that each have their own hiring accuracy.

The Talent Curve Simulator tables the shape of the Talent Curve for each of up to 10 stages in a multi-step staffing process. The Talent Curve is made up of five types of hires: Stars (Top 5% of the candidate pool), Achievers (80% to 95%), Keepers (60%-80%), Problems (40%-60%) and mistakes (Bottom 40% of the candidate pool). The simulator shows the percentage of each type as staff proceeds from self screening (based on the recruitment ad or website) through to qualification screening, culture matching, skill testing, performance assessment, and the final decision interviews.

After the Talent Curve Simulator averages the results from 100 hires, it converts the Talent Curve for traditional vs. advanced hiring methods into an average Z for the new hire distribution, which is then converted into an average dollar value using the Cronbach and Gleser formula.

The Personal Value Simulator

Corporations hire people assuming the actions they perform on the job result in more value than the wages, benefits, expenses, and energy they consume. Ideally, the hiring manager would like to be able to pull out a hand held Future Performance Value scanner (not unlike Dr. Spock’s tri-corder from Star Trek) that with an entry-level position dialed in and pointed at someone who wanted to work for the corporation, read out the expected dollar value of that person’s performance. Here is what such a readout could look like:

If the hiring manager could put all the candidates in a room and scan them, picking the right hires would be obvious—the ones with the highest Net Performance Value. Think of online Screening and Assessment as efficient ways to get a “readout” on those individuals motivated to want to work at a specific position opening. It can’t be in Average Net Performance Dollars, but the candidate fit score relates to such a dollar index according to the accuracy of the online assessment used.

Net values close to and even below zero are possible. These are individuals who want the job, but whose action patterns on the job deliver less output value than the costs of their salary, benefits, and allocated capital expenses. If such a scanner existed, calculating the performance value of different staffing methods would be easy—total up the net performance values (over some fixed period) for the hires made by the methods you want to compare.

This simple table shows that for processing 100 hires, an Online hiring system that deploys best practices in candidate qualification, screening, and assessment yields $1.4M of additional performance value.

The image below shows the annual personal performance value for 100 Monte Carlo simulated candidates (not hires) for a field Pharmaceutical Sales Rep openings. The average annual starting compensation is $80,000. The difference in the performance value between 85th percentile performers and average (50th percentile) performers is estimated at 1X annual salary, since these reps can impact over $500,000 per year in drug adoptions through the doctors they visit.

The traditional selection method included a resume sort and an unstructured one-on-one interview, with a population validity of .25. The online, best practice comparison includes online qualification screening, an online sales test battery with personality and ability

scales, and a behavioral interview, yielding a combined validity of .65.

Obviously, if you knew the value of a room full of candidates, you would pick those candidates with the ten highest values. We can only ever know the candidate score on a predictor of candidate value—such as a screening test, an ability, test, or an interview or assessment rating.

The highest valued candidate ($480,676) was picked by the online best practice method, and not by the traditional method. Values in green were picked exclusively by the online method. Values in blue were exclusively picked by the traditional method. Notice how they are uniformly lower in value. Candidates in yellow or appearing in a colored background were picked by both techniques—either on the same try (yellow text) or on different picks (colored background).

The box on the lower right reveals that for this run of the simulator through 100 candidates and 10 hires, the performance value of just 10 hires is $392,000 higher for the best practice selection method. Given these online methods actually cost from 20-40% less than the traditional ones, the financial reasons for adopting online best-practice selection are obvious.

The Talent Curve Simulator

The Personal Value simulator demonstrates which of 100 candidates a traditional vs. best practice selection system would hire. The Talent Curve simulator breaks down the shape of the Talent Curve in two ways.

First, it breaks the staffing process into the multiple steps that are always involved. There may be a candidate self-screen via a position preview, then a qualification screen followed by a testing stage leading to a final set of decision interviews. Rarely is anyone selected by one step or measurement.

Second, the Talent Curve Simulator breaks the Talent Distribution into 5 types of hires, based on their job performance.

Thus you need to know the steps of the staffing process, the number of position openings over the projection period, the candidate flows through each step, and the hiring accuracy of the methods associated with each step, in order to set up the Talent Curve Simulator.

Figure 2 below illustrates the setup for the Princeton Search Group, the largest franchisee from the Management Recruiters International (MRI) network. They plan to hire 25 account managers this year, and about 100 for the next two years.

The proposed online Talent Pipeline includes: [1] an online realistic job preview, [2] an initial viability screen and culture fit questionnaire, [3] an online behavioral interview to make sure only qualified candidates are referred on for online testing, [4] an short, online test battery with ability and personality components related to sales performance, and [5] on-site behavior-based interviews between the GM, key office team members, and the candidate.

Figure 2: The Talent Curve Simulator for Princeton Search Group internal hiring.

The comparative existing Talent Pipeline includes the steps shown above in the red box. Meta-analytic research establishes the low hiring accuracy of resume sorts and telephone screening interviews. Without the benefit of a behavior-based interview structure, the GM interviews score lower and low validity for the P3 measure drags down the estimate for those branches that go through the more extensive Caliper test—still largely a personality-based predictor.

Figure 3 shows the results of making 100 picks from 3000 simulated candidates. The best practices recommended have a dramatic effect on the shape of the Talent Curve as candidates progress through each step. The bottom line of the steps shows the type of talent projected to receive offers. Best practice selection delivers 14% more Stars, 7% more Achievers, and 16% fewer Mistakes. Translated to dollars, this represents an annual $27,300 improvement in sales performance per hire. A close look at the fully loaded costs of the two approaches reveals a $300 cost saving per hire using the online best practices, since much of the labor cost loaded into the traditional approach goes away.

Finally Figure 4 shows the same data tabled in Figure 3 as a graph. Any way you look at it, selection methods that are cost neutral and even positive that deliver performance value savings in the tens of thousands of dollars per hire are just about the best investment in profitability a company can make.

Figure 3: Talent Curve Simulator results from 100 Monte Carlo simulated hires.

Figure 4: Talent Curve Bar graphs for Current vs. Best Practice methods.

Conclusions

I have argued that staffing practitioners who want to influence executives to adopt selection methods that clearly contribute to profitability should avoid the traditional utility analyst’s mistake of simply computing the total dollar saving and counting on the size of the number to drive the executive towards adoption of the improved selection practice.

Instead, involve the executive, and that executive’s hiring managers, in setting up the parameters for a Talent Curve Simulation. Then show the impact on the Talent selected by each stage of the hiring process. Finally, convert the ultimate new hire Talent Curve into a dollar value, showing the average annual increase in performance value first.

Then, once that credible value is multiplied by the number of hires and the average tenure, the resulting total dollar saving, a very large, hard-to-swallow number, is a mere multiplication that the executive can duplicate on his pocket calculator.