~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Get Zeroed-In on Learning Measurement
Issue 3
Winter 2006
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In this issue
-- Putting "wheels" on your data
-- Benchmarks: DOD ISO SSCP CISSP ... IA Alphabet Soup
-- Featured Measure: Revenue per Employe
-- Case Study: Validating success at Global Knowledge
-- Opinion: Learning analytics versus learning effectiveness
Greetings! Get Zeroed-In on Learning Measurement is a quarterly newsletter devoted to learning and performance measurement and the people and processes surrounding it. Each issue contains feature articles, benchmarks, case studies, opinions, and upcoming events relating to learning measurement, learning analytics, and strategic reporting inside and outside the learning organization. | |
Putting “wheels” on your data ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To steal a line from Dr. Allison Rossett, long time Professor of Educational Technology at San Diego State University, “data, data everywhere and not a drop to drink”. Dr. Rossett led a discussion in early March at the 2006 Questionmark User Conference in San Francisco where she pointed out an overwhelming need to mobilize meaning from the data that organizations capture. Dr. Rossett further refers to this effort as “putting legs on your data”. Many organizations capture too much data and don’t do anything with it. It’s time to take a step back understand what’s the purpose of all that data and what it means at an aggregate level. In other words, how can an organization make better, faster decisions that result in real business value without having to cull through mountains of data? How can all that data be aggregated into meaningful metrics that provide context for those strategic decisions? Enter the executive dashboard. BusinessWeek calls the executive dashboard the "killer app" for businesses of all sizes — a "must-have" application "that is beginning to change the nature of management, from an intuitive art into more of a science." The recent BusinessWeek article further states “ The technology is particularly valuable to small companies…Managers can see key changes in their businesses almost instantaneously…and take quick, corrective action.” In the 1990’s, most executive dashboards were custom applications built internally by mostly large organizations. But today, with the ubiquity of the Internet and rapid integration technologies, executive dashboards are less expensive and easier to deploy than ever. Turnkey dashboards cost under $11,000 per year and can be up and running in as little as one week. And dashboards aren’t just tools for CEO’s anymore. They provide valuable information to all levels of management throughout all parts of the organization – sales, manufacturing, human resources, and learning! Keep in mind that just presenting a series of charts, graphs, and tables in an executive dashboard format may not entirely convey the information in a manner conducive to making strategic decisions. Your dashboard and underlying metrics must have context in how they align and reflect progress towards accomplishing your strategic goals and business imperatives. If we marry Dr. Rossett’s concepts with BusinessWeek’s article, we can replace the “legs” in our data mobilization strategy with “wheels”. It just seems like a faster way to make that data travel across the information superhighway! Read the complete BusinessWeek article “Giving the Boss the Big Picture” at : http://www.getzeroedin.com/resources/BW-Dashboard.pdf | |
Benchmarks: DOD ISO SSCP CCSP 8570.1 … IA Alphabet Soup ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
All these acronyms, but what do they all mean? There is a common theme between these bits of alphabet soup. As you probably know, DoD stands for Department of Defense. ISO stands for International Standards Organization, SSCP stands for Systems Security Certified Practitioner, CCSP stands for Certified Information Systems Security Professional, and finally IA stands for Information Assurance. The relationship is cyber security and it all boils down to DoD Directive 8570.1 This Directive establishes policy and assigns responsibilities for DoD Information Assurance (IA) training, certification, and workforce management. The crux of this initiative is that all DoD personnel (civilian, military, and contractor) must have, at a minimum, information awareness training. But it doesn’t end there. Privileged users and managers must be fully qualified, trained, and certified according to DoD baseline requirements. DoD has benchmarked and aligned its baseline requirements to industry recognized certifications, specifically SSCP, CCSP, A+, Network+, and many others. The exact certifications for each resource depend on the person’s category, level, and function. The value in benchmarking to the industry certifications is in eliminating the “must be invented here” syndrome and reducing training development, delivery, and certification costs. ISO, the International Organization for Standardization, has published a standard on conformity assessment with general requirements for bodies operating certification of persons. The Assistant Secretary of Defense for Networks and Information Integration, along with the DoD Chief Information Officer, are responsible for establishing metrics to monitor and validate compliance with this Directive as an element of mission readiness . It’s also a great opportunity to capture and compare metrics across the various DoD components such as compliance by category and level, cost of compliance, and speed to compliance. You can read the complete directive at: | |
Featured Measure: Revenue per Employee ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
What it means: An important ratio that looks at a company's sales in relation to the number of employees they have. This ratio is most useful when compared against other companies in the same industry. Ideally, a company wants the highest revenue per employee possible, as it denotes higher productivity. How it's measured: Total revenue divided by total number of employees How it’s used: The revenue per employee ratio provides a broad indication of how expensive a company is to run. It can be especially insightful when measuring the efficiency of businesses such as banks, retailers, consultants, software companies and media groups. "People businesses" lend themselves to the sales per employee ratio. Interpreting the ratio is fairly straightforward: companies with higher sales-per-employee figures are generally considered more efficient than those with lower figures. A higher sales-per-employee ratio indicates that the company can operate on low overhead costs, and therefore do more with less employees, which often translates into healthy profits. | |
Case Study: Validating success at Global Knowledge ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Global Knowledge is the largest privately held provider of training, enterprise learning services, and software solutions for information technology (IT) and management professionals. It should be no surprise that a learning organization of this magnitude employs a finely tuned process to ensure that each customer engagement is successful. The process, known as the SOLVE™ methodology, contains five key steps – Surround, Originate, Launch, Validate, and Evolve. SOLVE is the vehicle that gives Global Knowledge the ability to fully understand their customer’s unique learning needs within the context of their business goals, in turn allowing them to design and implement turnkey and custom measurable solutions. “ The most effective learning solutions involve a process that's continuously monitored and adjusted to meet specific measurement and evaluation criteria”, says Chris Gosk, vice president of e-learning delivery solutions at Global Knowledge. “Using the SOLVE methodology, we are able to ensure the customer’s training initiatives adapt and grow as their business strategies and training requirements evolve.” Using a series of analytical processes, Global Knowledge surrounds the customer’s business in order to approach the learning initiative from all sides - their corporate culture, business goals, and learner needs. The analysis enables their learning experts to originate or create tailored solutions, including custom content development, based on sound instructional principles. They launch the solution using a combination of modalities for the learners, environment, and timeframe, guided by a project manager who ensures quality implementation throughout. Global Knowledge validates the solution by measuring and evaluating the results along the way to ensure success. They present the success metrics in an executive dashboard both internally for Global Knowledge management, and externally for the customer. Lastly, they continually evolve the solution by providing ongoing fine-tuning and recommendations so that the customer always has access to the right resources and tools for continuous learning . You can learn more about Global Knowledge by visiting: | |
Opinion: More to learning analytics than learning effectiveness ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
After spending the last two weeks at various measurement conferences, I’m convinced that there is some confusion in the learning marketplace with respect to learning analytics and the process of measuring learning effectiveness. They seem to be referred to as one in the same, but they are not synonymous. Yes, learning analytics includes the analysis side of measuring learning effectiveness, but the learning analytics process encompasses much more than learning effectiveness which is rooted in Kirkpatrick’s 4 levels of evaluation, plus Phillips ROI model. Case in point, we can’t drive effective learning at all cost. I’ll be the first to say that the Philips ROI model analyzes costs of individual learning programs. But, what about the costs of operating the entire learning organization? It is here where learning analytics goes beyond learning effectiveness and targets operational efficiencies and best practices of learning organizations. Best in class corporate universities have been benchmarking each other on operational measures for years. Training magazine, a VNU Media publication, has been researching and publishing its annual Industry Report and Salary Survey for years. These efforts are prime examples of learning analytics! Josh Bersin, principal and founder of Bersin & Associates, and an expert in learning measurement and benchmarking, encourages organizations to look beyond learning effectiveness. Josh says “ As much as learning organizations are trying to change their image, they are still a cost center. If you continue to only measure effectiveness, in essence trying to prove your value to the organization, you will continue to be recognized as only a cost center.” In breaking the paradigm, organizations should be measuring delivery efficiency, delivery volume, speed to market, talent gaps, succession planning, alignment, and workforce performance. Bersin further says that if want to measure effectiveness, try measuring the effectiveness of your LMS investment. For content, organizations must begin to measure relevance, not just impact or effectiveness. Impact is fine for the old course model, but relevance is needed for the knowledge on demand model. Compliance is another area where learning analytics is being leveraged – cost of compliance, speed to compliance, periodicity, and audit response. Manufacturing and financial firms routinely analyze these numbers looking for ways to improve efficiencies and reduce risk. Isn’t compliance part of learning? Or is that just training? And that’s my opinion on learning analytics. - Chris Moore |
|
Pertinent Articles and Resources ... ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| |
Upcoming Events ... ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ email: cmoore@getzeroedin.com phone: (410) 242-6611 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
||