Interview with Learning Expert: Will Thalheimer

Measuring Learning Results

T/D: You've written a research in practice report called Measuring Learning Results which is an understanding of learning and focusing on measurement.  Can you tell us more about that?

Thalheimer: Over the years I've been immersed in learning and looking at the research on learning.  One of the things I noticed in our field is that there are some leverage points - things that affect the whole field, things that drive us to do what we do.  One of those is measurement.  How we measure affects how we practice instructional design.  So I'll share with you a couple of things that I found.  One thing is that smile sheets aren't good enough.

T/D: Right, don't we know that at a gut level?

Thalheimer: We may know it at a gut level, but we keep doing it. I did some research with the e-learning Guild and we looked at how it's the most popular method, after completion rate, for an e-learning program.  So they've done like a research meta-analysis on this, and they found that smile sheet ratings don't even correlate with learning results.

T/D: Interesting.

Thalheimer: They correlated like 0.09 - which is…

T/D: Negligible.

Thalheimer: Negligible.  Smile sheets don't correlate with on the job performance either.

T/D: What does your opinion have to do with your performance?  Nothing.

Thalheimer: Right, so that's intriguing.  So one of the things we're trying to do when we measure stuff is to get better.  There are really three reasons that you might want to measure learning - one is to support learning, to give the learners feedback.  The other thing is to improve the learning interventions we create.  So improve our learning designs.  The third thing is to prove our results to our organization; our certification, grading, that kind of thing.  So three main reasons there.  If we're going to improve our learning designs one of the things that we need to do is make sure that our learning metrics are predictive of people's actual performance.  If our smile sheets aren’t predictive, then they're no good at doing that, they're no good at giving us good information.

I've been playing with trying to improve my own smile sheet so I'm going to recommend a couple of things.  One - let's not ask people just overall questions.  Let's put in some real learning points that we wanted to get across, because people aren't very good at remembering everything after a session, right?  So if you ask them in general did you like it you're going to get general responses. Instead; this is one of the learning points that we talked about, how valuable was that to you?

Then I ask them - what's the likelihood that you're going to use this within the next two weeks?  I have from a zero to 100% in 10% increments.

T/D: That's brilliant.

Thalheimer: I also ask them - what's the probability in the next two weeks that you'll share this with one of your co-workers?  I also really focus on the open ended comments.  I find that those are really some of the most valuable feedback that you can get to improve your course.  I've gotten a lot of bad feedback too, but I've gotten some really good feedback on smile sheets that has told me - you really need to improve that exercise.  Or, have more stuff going on in the afternoon that's fun because people were tired - stuff like that.  Okay, you hear that over and over - you better change it.

Thalheimer: Another thing I do with smile sheets is I also like to follow up two weeks later.  I follow up and I do a couple of interesting things.  One is I ask the same question on the same six point scale about how valuable you thought the training was.  Now they've gone back to the work place.  They now really know how valuable it is.  So it gives a better anchoring.

I’ve learned a couple of other things from looking at measurement from a learning fundamental standpoint.  One thing is, if we measure learning right at the end of learning, what is it we are measuring?  We're measuring the learning intervention’s ability to create understanding.  Which is fine, that's good, but we're not measuring the learning intervention’s ability to minimize forgetting.  You have a full day session or you have an e-learning program and you measure people right away - everything is top of mind.

T/D: You should be able to recall it.

Thalheimer: You can easily recall it and not only that, but you feel confident that you can recall it.  So that's a really biased way to get information.  That's one of the reasons that make it a bad proxy for your ability to remember it on the job.The final mistake we make is to measure something in the same context that people learned it.  That context is going to remind them of what they learned and because of that they're going to get better results than if we measured them in a different context.  So we need to do one of two things - we either need to know that our results are biased because we're in the same context that they learned in.  Or we need to change the context and make it more like the real world context so that it's more realistic.  It's more predictive of that real world environment.

T/D: Great points! Thank you Will.

Will Thalheimer founded Work-Learning Research which is a consulting practice that helps clients build more effective learning interventions.  He’s been in the learning and performance field since 1985.  His professional focus is on bridging the gap between the research side and the practice side

The Key to Outstanding Learning Outcomes: Design Learning Processes

Designing training programs can be an arduous process if you take on the responsibility for designing all the content yourself. You can lighten your load and also achieve a much greater learning outcome by designing learning processes rather than learning content.

In this article we will share three ways to design learning processes.

1. Use real work.

Learners prefer to accomplish real work while they are in the learning process. So rather than create a contrived situation or a case study scenario that is “representative” of real life, instead have the learners work on real work tasks. For instance, a financial services firm wanted to teach its sales people to read financial statements in order to find cross-selling opportunities in their current client-base. Rather than teach the sales people how to read “generic” financial statements and then leave them to transfer that knowledge and skill to their own client's financial statements, the learning session required them to bring the annual report from two of their current clients. While they learned to read a financial statement (such as a cash flow statement or a profit and loss statement) the learners were working and reviewing the actual financial statements of their own clients. This not only resulted in a better understanding of the learning but it also resulted in the learners being able to have actionable findings be the end of the training class.

2. Create the learning in real time.

Rather than teach your learners a new concept or skill in a large block of time and then expect them to be able to transfer all of that information to their real work responsibilities, break the training up into smaller, actionable learning objectives and on-the-job tasks. That will allow them to implement their new knowledge and skills in smaller chunks and result in more successful implementation on the job.

For example, a sales organization was training their sales people to listen for cues from prospects to better gauge if they could ask for an appointment or not. After a period of time  in the classroom, in which the sales people/learners learned the 5 types of responses which would either open a door for them or not, they were then given an hour to return to their desks and make up to 10 phone calls from their personal prospect list; making notes about types of conversations they had and whether they were able to secure an appointment or not. This approach not only allowed the salespeople to accomplish some real work during the training process, but also created a rich discussion upon returning to the training room because they each had real-world experience implementing their new skills and were able to compare and contrast their outcomes and ask additional questions of the instructor.

If they had simply completed the training and then returned to the job to (attempt to) implement what they had learned, when would they have ever had those rich discussions that serve to cement the learning for adults?

3. Have the learners contribute the content.

A third sales organization wanted to teach overcoming objections in relation to a very complex product that their sales people sold. Rather than try to anticipate all the objections and give the sales people pat answers in reply, the training was designed to first solicit the “5 toughest objection you've encountered when attempting to sell xyz” and then a game was created, dividing the larger group into 3 teams and giving each team the opportunity to craft an appropriate rebuttal to the objection.

The learning process went like this

Team A stated an objection to Team B~Team B had a period of time to craft an appropriate response~Team C had the opportunity to challenge Team B's response~Team A chose what they thought to be the best response and awarded a point to either Team B or Team C accordingly.

This learning process continued in a round-robin style, with Team B next sharing an objection, Team C getting the first opportunity to reply, and Team A being given the opportunity to “challenge” team C’s response, until all of the 15 toughest objections in regard to that topic had been addressed.

This process allowed the learners to share their real-world problems and to get the best and the brightest to assist them with being better prepared the next time they heard that objection.

The next time you are attempting to design learning content take a step back and see if instead you are able to design a learning process that better assists the learners in working with and assimilating that content.

Learning processes can often lead to greater learning outcomes because the learners are more engaged with the content, identify with it more clearly, and have less trouble transferring what they learned in the classroom to what they do on the job. An added bonus, from a logistical standpoint, is that designing a learning process requires much less updating in the future should the content itself change. 

Interview with an Expert: Susan Boyd

How to Make Learning Stick in Online Courses

T/D: Susan, why is there the perception that e-learning doesn't stick and what can we do about it?

Boyd: I think there are a variety of challenges that can cause an e-learning program to fail.  One in particular would be technology barriers which result in the trainee/learner not having the proper access to get to the course. Often a trainee/learner gets to the course only to find the content is really not pertinent to their job needs or find they have to go through 100 page turning screens just to find a small nugget of content that is pertinent.  It’s quite possible  the training will not stick because it's almost as though we're hiding the forest with the trees.  Trainees are not getting the overall picture.

Some trainees/learners really just want to get through the content thinking the ultimate purpose to e-learning is to get to the end. Many are only looking to find the “next” button and continue to push it until it’s at the end of the session.

T/D: What are some tips that can make this better?  Are your tips coming from the perspective of instructional designers who are the ones who create the training or are there tips for us as learners?

Boyd: Most of my tips have to do with content - I think that's the instructor and designer.  How to break up the content into chunks of learning, how to make it as job specific as possible.  Try to give the learner control of the content, make it interactive - those are things that instructional designers can do.

Preparing the learners and preparing the managers are also really key.  Many managers thought e-learning was just this magic button people push on their computer and, through osmosis, people would instantly learn the topic.  Managers don’t often invest enough time in following through:  Did people do it?  Do they understand it?  And are they applying it?

The key thing for the learners is to ensure, they can access it.  We have a generation of learners today that are more internet savvy - so that's wonderful.  But their computers at home are typically better than their computers in the office.

We need to make sure we're preparing them by whatever wizards they need to test their system.  Whether it's a webinar they're going to attend or it's how to get to the learning management system to enroll in the online course.  It needs to be made as easy as possible.  Maybe some frequently asked questions, or getting started guides are helpful.

T/D:    That’s not  as much making the learning stick as making the environment more accessible for the learning to stick.

Boyd: Right. Learning can't stick if you can't get to the course. Good point.  I look at this as a multi-faceted problem.  Part of it is the content design with the instructional design; part is working with the learners; part is working with the managers - providing them the right support material for what they need to do.

In terms of making learning stick, it’s important to make it real world.  People can relate to it and find the information in the course, but can also find it outside the course.  If there's a reference card or frequently asked question which can only be found in the course, that becomes the challenge because these are the types of things people need to access outside the course.  So there ought to be a way that they know where to find those same types of online references.

T/D: What tips do we have for training designers then?

Boyd: I think it's engaging and having the learning relate to something - hopefully it's a real world example.  Making some connection ia key part of it.  Using graphics and pictures - for example in safety courses - I worked with one client and the pictures we use are pictures of people on the factory line.  This particular client is a manufacturing company and there are examples of them where something is unsafe.  Maybe they're not wearing the proper protective equipment or there's a tripping hazard.  You're taking these real pictures and saying, what's wrong?

T/D: That's fun!

Boyd: Or saying, there are six things wrong with this picture, can you define them or click them.  In that case defining them sometimes is actually click in the area where you think there's a problem.  So it is forcing the learner to do something – we want the learners to do more than click on the “next” button.

TD: What I hear you saying, is in order to make e-learning stick, you have to understand how the learner is  going to use the content and support them through the process.  It's about managing the manager's expectations on how the learning will  be used or what the end result of the learning is.  Then designing it so that it's mentally stimulating and engaging for the  participants, which is a design process. It's pulling the learning from the participants which is what we normally do in the classroom, but we don't think it can be done online in my estimation.

Boyd: That's the challenge. E-learning was supposed to be able to get the right content to the right person at the right time.  They were equations that we had trouble with in the classroom - we could never deal with the right place and time - e-learning kind of takes away that barrier.  But we're still struggling with the content.

The design of the content has to understand the environment in which the content is going to be used.

T/D: Thank you Susan, The most powerful thing you said to me today was that when we take the learners out of the equation, we design bad e-learning.  That's really powerful because I think too often we think of e-learning as the 'e' part and we have to start thinking about it as the 'learning' part.

= = = = = =

Susan Boyd has over 30 years' experience in training and computer education, with extensive experience in planning and managing computer training projects. She has directed and participated in all phases of the projects including needs analysis, course development, train the trainers, course delivery and follow-up.  Susan has developed and delivered training in a variety of formats including instructor-led classroom, virtual training sessions using a webinar, and online asynchronous e-learning courses. You can learn more about Susan and contact her at www.susan-boyd.com

Training Triage – Pick One

One of the things The Training Doctor specializes in is TRAINING TRIAGE - that is - why doesn't training work? Throughout our 20+ years in business, we've discovered the answers to this question. This month, we share with you one of the reasons:

PICK ONE:

Very often, especially when teaching a psychomotor skill (how to manipulate something) there is more than one way to complete the task. For instance, when copying text from a Word document in order to insert it somewhere else - one can use the keyboard (Ctrl+C), the ribbon (Home>Copy) or right click and choose from the drop-down-menu. While the variety of options makes for a very user-friendly software it does NOT make it learner-friendly. When something is new to an individual, it must be taught in one way and one way only; offering multiple techniques only leads to confusion and a lack of mastery of any one.

Two solutions for solving this common problem are:

1. Officially choose the ONE way to be taught and

2. Document the one way in your learning materials - both facilitator materials and participant materials.

The latter solution is particularly important because we all have our "favorite way" of doing something, so it is imperative that all trainers understand, that for the sake of learning, the ONLY way that can be taught is the documented way. It is OK for the trainer to say "There is another way of completing this task, and once you have mastered THIS way, we can teach you the others," but do NOT allow them to say "I know it says XYZ in your participant guide, but let me show you an easier way."

Imagine that you had three driver's ed instructors when you were learning to drive - one in the front seat and two in the back seat all shouting out different ways to approach an intersection. Would you have mastered any of them? Would you have remembered any one of them? The key to efficient learning is to teach only one way. The others will come on-the-job through informal learning, or can be documented in an appendix for the 'advanced' learner.

Quotable: Bob Pike

Instructor-led, participant-centered training is about involving participants in every way possible in the learning process. The more participants are involved in the content, the greater the retention and application.

Quotable: Bob Pike, Creative Training Techniuqes

Incentivize your training - a great model

On September 29 a concert was held in New York City's Central Park to bring awareness to worldwide issues such as disease, poverty and lack of drinking water. 

The concert was free and 60,000 people attended BUT they had to earn the right to attend. First, they had to register at a website. Then, they had to earn points to get in to a lottery to be awarded the free tickets. They earned points by watching various videos on the issues above and / or then forwarding those important messages to their friends via Facebook or Twitter.

What a GREAT model for making your training viral! Especially when you are constrained by having to disseminate your learning through asynchronous methods (in other words, people will engage in the learning on their own time and schedule).  Why not have prizes or awards for completing the training and certain tasks along the way?

If you'd like to read more about the concert, click here

Seeing is.... comprehending

You know the phrase "seeing is believing?" There is much truth to that, it turns out. However, in our world, the phrase should be, "seeing is comprehending."

Although it is our eyes that take in visual stimuli, it is our brains that make sense of it. John Medina has a chapter (#10) in his book, Brain Rules, which includes some interesting examples of patients who have perfect sight but damage to their brains, in one way or another, that prevents them from understanding or interpreting what they see.

The brain processes shapes and symbols by putting them together in an 'organized' way and thereby making sense of them and making a connection to them. The brain works hard to make these connections.  One of the reasons optical illusions work so well is because they purposefully interrupt the making of connections.

What is the lesson to be learned here in terms of training? Only one that we have been trumpeting for years: use visuals in your learning materials.  The brain will naturally associate the content with the visual. It helps to have visuals that link to the content (e.g. Here are the 5 keys to qualifying a sales prospect, accompanied by a photo of a key) however, they don't have to. Any visual will cause an associated link in the learner's mind. You could simply have the 5 bullets set off in a colorful table and that would make a visual connection to the words for the learner, as well.

Visuals belong in the obvious places (your slides) and the not so obvious places (the workbook, the job aid).  Our visual cortex is the most developed of all our senses so you use it to your advantage in developing training.

Employee training leads to strong business performance

The #1 focus of Hubert Joly, Best Buy's new CEO, is employee training. He believes that inconsistent training across stores is what has led to uneven performance among stores and overall sales decline. His mission is to make store employees an "undisputed point of reference" for customers. 

Source: Chicago Tribune

Quotable: Peter Casebow

"Some would say you can't control or plan for something like informal learning, but you can put a strategy in place. Based on our experience, any strategy for informal learning needs to include three basic areas: improving basic skills, such as searching for information effectively, creating opportunities and encouraging sharing and collaboration."

Quotable: Peter Casebow, CEO of GoodPractice

Why the 2-hour training class doesn't work

Research on what’s known as the “spacing effect” shows that we form stronger and more lasting memories by exposing ourselves to information over time. Repeated cycles of learning, consolidating, and then re-encountering material fix it firmly in our minds.

When does the learning occur?

Let’s preface this article with two assumptions:

1. Most people who design training are not schooled in how to design training, these days. They are more typically subject matter experts.

2. Most people who ARE schooled in training still are not schooled in adult learning theory.

Over the years, we have noticed a recurring fault: designers (and facilitators) of training often equate participating in an activity with actually learning from it.  Not so.  Here are two examples:

• A course in sales training, that we designed  for a client, required a breakout group activity. At the end of the activity, each group was to present back their outcomes.  As each group finished their presentation the facilitator replied with “good dialogue, good dialogue” and then moved on to the next group.  The facilitator spent NO time asking for further interpretation of their decisions or outcomes, he spent NO time comparing one group’s response to another, he never asked any other group to respond to or ask questions of the presenting group.

• A training design that we reviewed for a client included a 10-minute, independent, web browsing activity in which learners were to research the “key selling features” of a particular product. Once the activity was over, the course moved on to the next topic. There was no discussion of what people had found. There was no compare and contrast. There was no knowing if you did the activity right or not!

So the question we pose is this: When does the learning occur?

Lucky for you, we’ll give you the answer as well: It occurs in the debrief. The debrief happens after the activity. Rarely, if EVER, do participants get the ah-ha moment from simply participating in an activity. Ninety-nine percent of the time it occurs during the debrief when they are asked to process, sort and supply their findings. When they hear what others, or other groups, have come up with and compare it to their own. When they are asked “so what does this mean for you, on the job?”

Simply designing or facilitating an activity is not sufficient.  The REAL learning comes after the activity.  Discussions with peers, working collaboratively or competitively, all help to ensure that learners actually process what has just occurred and gain meaning from it.

Ensure that you think that far ahead when you design for your learners.

Speed to Mastery - from Forum Corporation

Forum Corporation has designed a salesforce.com-like product for training. It combines classroom training, individualized support for when learners are back on the job, and a game-like interface.  It’s an innovative and inspiring way to learn and ensure on-the-job skills application!

You can learn more here: https://vimeo.com/44737652

Note: Forum Corp is one of The Training Doctor’s clients but we aren’t promoting Speed to Mastery because of that – we are promoting it because it is BRILLIANT.

Look BEYOND the training - if you want it to be successful

According to Robert Brinkerhoff, training events alone typically result in only 15% of transfer of learning to on the job behavior. So if you truly want your participants to be successful on the job, after training, you need to think beyond the training event itself.

There must be processes or systems in place that reinforce, monitor, encourage , or reward the performance of those things you consider to be critical on the job behaviors. We spend much of our time as trainers, worried about Level 1 and Level 2 outcomes (did the trainees like the training in the short-term and did the trainees leave with more knowledge than they came with) but not enough time on whether or not the trainees are implementing their new skills and knowledge on the job.

Before you start any training program, start with the end in mind, because the training will only contribute 15% to the success of your initiative. Be especially analytical of what you expect to see people doing differently on the job and how you expect them to be successful on the job. Very seldom will someone have the initiative or the time or the thorough understanding to be able to transfer what they learned in a class to their real work responsibilities.

Delta Takes Training Evaluation to the 4th Degree

Delta flight operations training goes through level three evaluations for "everything we do," says Scott Nutter, Flight Operations General Manager, and level four evaluation by tying training data to operational performance and safety metrics.  

As a result, Delta has received several awards including Travel Weekly's Magellan Award and being named to Fortune's World's Most Admired Companies in 2011.

Delta and Northwest merged more than three years ago and had a smooth integration of their training initiatives by keeping focused on these important level 3 and level 4 outcomes.

March /  April 2012 Training Magazinep. 40