~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Get Zeroed-In on Learning Measurement

Issue 6

December 2006

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In this issue

-- Training impact evaluation that senior managers believe and use

-- Benchmarks: Most important measurement initiatives

-- Featured Measure: Hiring “value proposition” index

-- Case Study: Cash America International rings up value in learning

-- New book spotlights digital dashboards as learning planners

Merry Christmas, Happy Hanukkah, and Holiday Cheer from Zeroed-In Technologies!

Get Zeroed-In on Learning Measurement is a quarterly newsletter devoted to learning and performance measurement and the people and processes surrounding it. Each issue contains feature articles, benchmarks, case studies, opinions, and upcoming events relating to learning measurement, learning analytics, and strategic reporting inside and outside the learning organization.

Training impact evaluation that senior managers believe and use

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Training professionals always talk about measuring the impact that training has on the business. But if you stop and think about it, training really doesn’t produce impact, so how can you measure the impact it has? Training produces capabilities. The business must apply the capabilities to produce some type of impact. The thinking that training produces impact seems to lead training professionals to want to evaluate training when perhaps they ought to be evaluating how the business is using training. What we end up with is a measure of how and if training is being used. And if it’s not, why not?

Dr. Robert Brinkerhoff, Professor Emeritus, Western Michigan University, has developed a training impact evaluation model called “The Success Case Evaluation Method”. This model does an outstanding job of measuring how training is being used. It starts by asking attendees a simple questions with three possible responses some period after the training occurs:

Q. Describe the nature of how you applied the knowledge, skills, and/or abilities you learned in the recent [fill in the blank] training class?

A-1. Did not try it at all

A-2. Tried it, had trouble or it didn’t work, so gave up and went back to old way.

A-3. Used it and achieved a concrete result that is a) worthwhile to the business; and b) could be documented with evidence.

The results from this evaluation are fairly predictable. Brinkerhoff says that 15% of respondents never try, while another 15% tried and got positive results. This leaves an astonishing 70% in the middle that tried it to some extent but got no worthwhile results. On average, that’s an 85% scrap rate for training if you include those that don’t try it at all!

The Success Case Evaluation Method goes a step further by isolating failure reasons that contribute to the non-uses of the previously learned knowledge, skills, and abilities. There are three specific areas that contribute: the learner’s preparation and readiness; the learning intervention; and the application environment. On average 20% of respondents say that the learning intervention was inadequate. Kudos for training, right? 40% say the root cause was preparation and readiness (e.g. wrong people attended, no clear reason for attending, didn’t need the skills, etc.) while the other 40% said the application environment was the cause (e.g. didn’t get any manager support, had no opportunity to try it, no incentive to use it, etc.).

The next objective of the exercise is to do the reverse with the original 15% that had documented successes and determine what of the three areas above contributed to their success most. By bringing the two results together, the training organization can focus on influencing the parts of the business, typically outside of learning, that can offer the greatest impact on helping workers apply their new skills that in turn produce positive business results.

This model produces immediate results for training by developing credible and documented stories of success. It also provides knowledge of the factors that drive impact. In other words, it uncovers the enablers of learning (e.g. manager support, special opportunities, etc.). Other benefits and applications of the Success Case Evaluation Method include: evidence that training works; information to educate customers of training; marketing; justifying budgets; designing impact enhancement procedures; redesigning training processes, and more.

Benchmarks: Most important measurement initiatives

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In a recent poll of visitors to Zeroed-In website (www.getzeroedin.com), the following question was posed: “Which of these initiatives is most important to your organization in terms of measuring its success?”

Here is how the responses stack up:

Talent management ………………….37.5%

Learning impact ………………………..50%

HR Value Contribution ………….…… 0%

Operational Efficiency ……….….…… 0%

Leadership Development ….……...12.5%

Workforce Transformation ….….…. 0%

As the results show, there is a renewed or perhaps just new focus on measuring talent. The industry has surely picked up on this and is helping to drive it through media publications like Talent Management (formerly Workforce Performance). Look for an upcoming article in the February 2007 issue entitled “Talent Measurement: A Management Necessity” by Chris Moore, President, Zeroed-In Technologies, LLC.

To view the complete results of this poll and to participate in our new poll, please visit http://www.getzeroedin.com

Featured Measure: Hiring “value proposition” index

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What it means: The organization successfully articulates its vision, value, and career opportunities to candidates and applicants during the hiring process.

How it's measured: % of candidates and applicants from the initial qualifying stage that elect to move forward in the hiring and selection process

How it’s used: The hiring “value proposition” index is a talent management measure used to gauge the success at which the organization keeps qualified candidates interested, engaged, and desiring a job position with their firm.

Case Study: Cash America International rings up value in learning

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Cash America International, Inc. is a diversified specialty finance company serving the needs of the underbanked segment of the population. They are the market leader in secured non-recourse lending to individuals, commonly known as pawn loans. Cash America operates over 450 pawnshops in the United States under the brand names Cash America Pawn and SuperPawn and provides other financial services such as check cashing and short-term cash advances through its network of lending locations and subsidiary companies.

Cash America operates in a highly regulated industry and the various business segments rely heavily on the efforts of Cash America Training to provide their coworkers with the knowledge and skills required to succeed. When it proved time to raise the visibility and accessibility of their progress measures towards delivering outstanding learning services, increasing time to competency, establishing workforce readiness, and meeting compliance, Cash America Training partnered with Zeroed-In Technologies to implement CLO Dashboard.

The week long implementation was split into two components, strategy and tactics. A full day strategy workshop helped Cash America Training articulate their learning strategy in terms of high level goals and learning outcomes also known as key learning indicators. The workshop deduced each goal into a series of objective-level strategies and key metrics, targets, and timelines designed to measure progress in accomplishing the strategy points and reaching each goal. The workshop process tests each key measure for accuracy, accessibility, and efficiency by identifying and analyzing the underlying source of data of which the most common source was the Cash America Train LearnCenter hosted by Learn.com.

Installing the web-based dashboard at Cash America in Ft. Worth, Texas and inputting the Cash America Train scorecard as determined during the workshop rounded out the tactical component of the implementation. After connecting the system to the Learn.com database to automatically extract measurement data on a recurring basis within the course of a few days, Cash America leaders had visibility and insight into the alignment strategy, success metrics, and progress of Cash America Training.

What does Cash America Training have to say about their experience with Zeroed-In and CLO Dashboard? Sandy Fulton, Vice President, Cash America Training says “Before CLO Dashboard, we had been through many iterations of learning measurement and reporting initiatives in attempts to give our stakeholders the information they need to make better decisions and investments. With CLO Dashboard, we now present a full picture of value creation for those stakeholders and an easy to use site for them to measure our progress towards serving our coworkers and growing our business. We are delighted with the speed, quality, and affordability by which Zeroed-In has brought CLO Dashboard to Cash America.”

New book spotlights digital dashboards as learning planners

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Dr. Allison Rossett, long time Professor of Educational Technology at San Diego State University, and co-author Lisa Schafer enlighten us on practical job aids, performance support, and measurement tools like CLO Dashboard™ in their new book "Job Aids & Performance Support: Moving from Knowledge in the Classroom to Knowledge Everywhere". In the book, the concept of Planners is introduced as tools and methods to support performance just before or after a challenge is being targeted by learning. Rossett and Schafer say “they [Planners] provide information in advance of performance so that expectations, requirements, or goals are set and then met. They provide information afterward to enable reflection and planning for future improvement.” In this intriguing chapter, Examples of Planners Performance Support, the pair goes a step further to describe five Planner support tools, including CLO Dashboard, by providing commentary about the original need and what was done to meet that need. In wrapping up the chapter, Rossett and Schafer highlight that “Planners, such as dashboards, make information accessible to decision-makers. They also encourage collaboration and are tied to the planning that goes into achieving business objectives.”

Here’s what some of your peers and thought leaders in the industry are saying about it ….

"Job Aids and Performance Support in the Workplace gives us everything we've ever wanted to know about these invaluable tools and techniques! Allison Rossett and Lisa Schafer have created a comprehensive, pragmatic, and very readable guide. The authors don't exaggerate when they claim it's 'knowledge everywhere.' "
—Pat Crull, chief learning officer, Time-Warner

"Allison Rossett shines her spotlight on performance support through job aids, low and high tech, available now and coming soon. She and co-author Lisa Schafer dazzle us with possibilities for Planners (for just before and after the performance challenge) and Sidekicks (great term! for what to do right now). Lively language, wide-ranging examples, and cautions about possible dark sides make this book an invaluable sidekick for performance-focused professionals."
—Mary L. Broad, international consultant and author, Beyond Transfer of Training

You can read more about this book online as well as purchase it at Amazon.com.

http://www.amazon.com/Job-Aids-Performance-Support-Everywhere/dp/0787976210

Pertinent Articles and Resources ...

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

· Previous Get Zeroed-In on Learning Measurement newsletters

· Seven key learning indicators your CEO really needs to know

· Measuring Success: Capturing the right metrics

· Measuring effectiveness with learning analytics

· Using models to manage strategic learning investments

· CLO Dashboard - Performance scorecard and dashboard for learning

Upcoming Events ...

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~