~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Get Zeroed-In on Learning Measurement
Issue 5
Summer 2006
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

In this issue ...

Get Zeroed-In on Learning Measurement is a quarterly newsletter devoted to learning and performance measurement and the people and processes surrounding it. Each issue contains feature articles, benchmarks, case studies, opinions, and upcoming events relating to learning measurement, learning analytics, and strategic reporting inside and outside the learning organization.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Innovative Leadership and Measurement
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Innovation and leadership go hand and hand.  It is the innovative leader, the one who thrives on change and is not afraid to take risks, that has helped set many of today’s leading organizations apart.  Where traditional leaders endorse innovation, innovative leaders see it as the only way to get a lead in the industry and keep it. They move fast to grab opportunities before others do. They rarely stop, and they rarely look back. Take 3M for example, a highly innovative company.  Their slogan is “Never kill an idea, just deflect it”.  When it comes to measuring innovation, 3M sets an annual innovation quota for its leadership team (i.e. percent of their products that have to be new).

Like their traditional counterparts, innovative leaders are measured on the financial aspects of the organizations they lead as well.  Financial measures typically show a lagging or coincident indication of success.  But because innovative leaders live more in the future than in the present, like at 3M, innovative leaders are measured on how many ideas from throughout the organization are implemented.  After all, creativity is the act of coming up with useful ideas, innovation is implementing and commercializing them.

Just as the Balanced Scorecard model integrates a series of processes and measures that follow a value chain of success for the organization, so does the measurement model for innovation and leadership. Since innovative leaders truly view new, fresh, and breakthrough ideas as the lifeblood of a healthy and growing organization, they focus and are evaluated on how they help get the ideas implemented that improve quality of life for employees and customers as well as the operational aspects such as productivity, efficiency, effectiveness, and product quality, all of which ultimately affect financial performance.

To learn more about innovative leaders, check out the recent article from the June 2006 issue of CLO Magazine at http://www.clomedia.com/content/templates/clo_article.asp?articleid=1397&zoneid=25

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Benchmarks:  Learning dashboard and scorecard initiatives
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

At the beginning of June 2006, Zeroed-In posed the following question to its website visitors using an automated polling and response utility: “What is the status of your learning and performance measurement initiative?”

Our intent was to determine the current progress that learning organizations are making with deploying automated dashboards and scorecard systems for measuring learning and performance initiatives.  Here is what we found:

    Fully automated and aligned dashboard or scorecard …… 9%
    Have dashboard or scorecard but mostly manual to collect data …… 9%
    Just starting a dashboard or scorecard initiative ……… 55%
    Have not even thought about it ……… 27%

To view the complete results and to participate in our new poll on important measurement initiatives, please visit http://www.getzeroedin.com

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Featured Measure: Percent (%) of job positions filled internally
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

What it means:  The fall out rate of learners in key programs, typically new hires that enroll and begin specific learning programs (e.g. orientation) but do not exit the on-boarding process normally or successfully.

How it's measured:  # of requisitioned jobs filled by personnel that were previously holding another job position within the company (e.g. non new hires); divided by the total # of requisitioned jobs filled.

How it’s used:  Percent (%) of job positions filled internally is used for managing and improving both talent management and succession planning programs and processes within an organization.  Those organizations with a higher ratio of job positions filled internally than those filled externally through the new hire process are identified as having strong talent management and succession planning initiatives.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Case Study: JetBlue University files in formation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

JetBlue Airways Corporation has built a reputation for outstanding customer service.  Since it took to the air in February 2000, JetBlue has instilled its core values: Safety, Caring, Integrity, Fun and Passion into every part of its business. These core values form the foundation for a set of pillars on which the success of JetBlue stands.  These pillars, collectively called their 3P’s, are People, Prosperity, and Performance.  Each year the leadership at JetBlue examines their efforts surrounding the 3 P’s and maps out their corporate flight plan, a strategic roadmap that aligns their most important goals, objectives, and underlying measures of success to effectively drive their people, prosperity, and performance. One of the leaders participating in these sessions is Mike Barger, Vice President and Chief Learning Officer, of JetBlue University.

Alignment and measurement has become a cornerstone at JetBlue University. Traditionally focused exclusively on learning effectiveness, Crewmember Experience and Impact, the team charged with measuring and reporting on the success and outcomes of JetBlue University learning initiatives, is now following the corporate strategy model.  By thinking of the 3P’s as a set of key learning and performance indicators, they are defining and implementing JetBlue University’s strategic learning goals, enabling strategies, and progress measures in an effort to monitor, align, influence, and inevitably reach the destination as charted by the corporate flight plan.  Under the direction of Shawn Overcast, Manager, Learning Technologies, JetBlue University is piloting a measurement scorecard approach that allows them to gauge the health of their initiatives at various levels.  Their approach combines proven business concepts, like the Balanced Scorecard and Oregon Productivity Matrix, with learning measurement practices, so they report and monitor in a manner the entire corporation, not just learning professionals, can understand.  “We’re excited to offer our internal customers and stakeholders this type of insight.” says Overcast.  “Showing our alignment and value to them will foster stronger relationships and new opportunities for JetBlue University to positively impact the business.”

JetBlue University is the central support organization for a series of operational-level colleges that provide Crewmembers with the critical training they need to perform their jobs. By defining cascading goals and individual scorecards for each college, and providing access through a JetBlue University dashboard, Barger envisions a perfect formation of the training functions at JetBlue, a sight he can relate to as a former US Navy Top Gun instructor.  As the learning leadership at JetBlue steps into the proverbial cockpit each day, their learning dashboard instrument panel can guide them to the key areas that need attention to collectively stay the course on the corporate flight plan.  And if they arrive early at their destination, they can be certain it wasn’t just the tailwinds that made it all possible. 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Opinion: Learning quality index -vs- Overall learner reaction score
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When it comes to determining the quality of your learning activities, how much do you rely on your overall learner reaction scores?  Donald Kirkpatrick’s four levels of evaluation define Level 1 – Reaction as measuring the participant’s perception of the learning activity.  I’m sure you’ve heard the statement that “perception is everything”.  Well in this case, I tend to agree with Kirkpatrick.  Learning departments generally ask a myriad of questions ranging from content relevance to instructor preparedness to environmental conditions.  All of these questions and the responses they solicit are intended to help us improve our learning assets, and, in most cases, the learning experiences.  I believe the overall learner reaction score from all of the questions that are asked can give us a good indication of the learner’s satisfaction albeit if the right questions are asked.  But can we depend on the same overall average score as an indication of the quality of our learning programs?  I say no, and my colleague Dr. Merrill Anderson, CEO of MetrixGlobal LLC and the 2003 ASTD ROI Practitioner of the Year, agrees.

Dr. Anderson’s research and findings on Kirkpatrick Level I evaluations tells us that there are actually just three (3) specific questions which when all weighted equally (0.3334) and averaged will form a true learning quality index.  All the other questions on your Level I survey are basically irrelevant when it comes to determining learning quality.  Now it’s not to say that those other questions may not be important to you and your initiative in some way. But as far as learning quality goes, it’s simply a factor of three.

So with your curiosity peaked, I bet you really want to know what those three questions are!  Well, I’m going to tell you with no obligation on your part … except one.   The digital age has really decreased the amount of human interaction between professionals, so in order to get those three critical questions I’m going to buck the trend and request that you pick up the phone and simply call me … (410) 242-6611.

And that’s my opinion …

-- Chris Moore 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Recent News Items ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
• Zeroed-In earns "2006 - Top Specialized Learning Process Provider" award
• Zeroed-In and Metrix Global parter to provide measureable business cases
• Zeroed-In releases CLO Dashboard v2.0 for learning governance

Additional news items available at: http://www.getzeroedin.com/press.php 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Pertinent Articles and Resources ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 •  Previous Get Zeroed-In on Learning Measurement newsletters
 •  Seven key learning indicators your CEO really needs to know
 •  Measuring Success: Capturing the right metrics
 •  Measuring effectiveness with learning analytics
 •  Using models to manage strategic learning investments
 •  CLO Dashboard - Performance scorecard and dashboard for learning

Additional resources available at: http://www.getzeroedin.com/resources.php  

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Upcoming Events ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
• November 5-8, 2006 – Learning 2006 (hosted by Elliott Masie) – Orlando, FL
• November 13-16, 2006 – Corporate University Week – Orlando, FL
• February 8-9, 2007 – The Conference Boards – Enterprise Learning Strategies Conference – New York, NY

Details about these events available at: http://www.getzeroedin.com/events.php

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Contact: Chris Moore, Zeroed-In Technologies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
email: cmoore@getzeroedin.com
phone: 410.242.6611   
web: http://www.getzeroedin.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~