Interview with Will Thalheimer, PhD

Thalheimer (1)

What motivated you to write this book?

I've worried about my own smile sheets (aka response forms, reaction forms, level 1's) for years! I know they're not completely worthless because I got useful feedback when I was a mediocre leadership trainer-feedback that helped me get better.

But I've also seen the research (two meta-analyses covering over 150 scientific studies) showing that smile sheets are NOT correlated with learning results-that is, smile sheets don't tell us anything about learning! I also saw clients-chief learning officers and other learning executives-completely paralyzed by their organizations' smile-sheet results. They knew their training was largely ineffective, but they couldn't get any impetus for change because the smile-sheet results seemed fine. 

So I asked myself, should we throw out our smile sheets or is it possible to improve them? I concluded that organizations would use smile sheets anyway, so we had to try to improve them. I wrote the book after figuring out how smile sheets could be improved.

If you could distill your message down to just one - what would it be?

Smile sheets should (1) draw from the wisdom distilled from the science-of-learning findings, and (2) smile-sheet questions ought to be designed to (2a) support learners in making more precise smile-sheet decisions and (2b) should produce results that are clear and actionable. Too often we use smile sheets to produce a singular score for our courses. "My course is a 4.1!" But these sorts of numerical averages leave everyone wondering what to do. 

How can trainers use this book to assist them in the work that they do?

Organizations, and learning-and-development professionals in particular, can use my book to gain wisdom about the limitations of their current evaluation approaches. They can review almost 30 candidate questions to consider utilizing in their own smile sheets. They can learn how to persuade others in using this radical new approach to smile-sheet design. Finally, they can use the book to give them the confidence and impetus to finally make improvements in their smile-sheet designs-improvements that will enable them to create a virtuous cycle of continuous improvement in terms of their learning designs.

Getting valid feedback is the key to any improvement. My book is designed to help organizations get better feedback on their learning results. 

Do you have a personal motto that you live by?

Be open to improvement. Look for the best sources of information-look to scientific research in particular to enable practical improvements. Be careful. Don't take the research at face value. Instead, understand it in relation to other research sources and, most importantly, utilize the research from a practical perspective. 

Will Thalheimer, PhD, PresidentWork-Learning Research, Inc.

Interview with Learning Expert: Will Thalheimer

Measuring Learning Results

T/D: You've written a research in practice report called Measuring Learning Results which is an understanding of learning and focusing on measurement.  Can you tell us more about that?

Thalheimer: Over the years I've been immersed in learning and looking at the research on learning.  One of the things I noticed in our field is that there are some leverage points - things that affect the whole field, things that drive us to do what we do.  One of those is measurement.  How we measure affects how we practice instructional design.  So I'll share with you a couple of things that I found.  One thing is that smile sheets aren't good enough.

T/D: Right, don't we know that at a gut level?

Thalheimer: We may know it at a gut level, but we keep doing it. I did some research with the e-learning Guild and we looked at how it's the most popular method, after completion rate, for an e-learning program.  So they've done like a research meta-analysis on this, and they found that smile sheet ratings don't even correlate with learning results.

T/D: Interesting.

Thalheimer: They correlated like 0.09 - which is…

T/D: Negligible.

Thalheimer: Negligible.  Smile sheets don't correlate with on the job performance either.

T/D: What does your opinion have to do with your performance?  Nothing.

Thalheimer: Right, so that's intriguing.  So one of the things we're trying to do when we measure stuff is to get better.  There are really three reasons that you might want to measure learning - one is to support learning, to give the learners feedback.  The other thing is to improve the learning interventions we create.  So improve our learning designs.  The third thing is to prove our results to our organization; our certification, grading, that kind of thing.  So three main reasons there.  If we're going to improve our learning designs one of the things that we need to do is make sure that our learning metrics are predictive of people's actual performance.  If our smile sheets aren’t predictive, then they're no good at doing that, they're no good at giving us good information.

I've been playing with trying to improve my own smile sheet so I'm going to recommend a couple of things.  One - let's not ask people just overall questions.  Let's put in some real learning points that we wanted to get across, because people aren't very good at remembering everything after a session, right?  So if you ask them in general did you like it you're going to get general responses. Instead; this is one of the learning points that we talked about, how valuable was that to you?

Then I ask them - what's the likelihood that you're going to use this within the next two weeks?  I have from a zero to 100% in 10% increments.

T/D: That's brilliant.

Thalheimer: I also ask them - what's the probability in the next two weeks that you'll share this with one of your co-workers?  I also really focus on the open ended comments.  I find that those are really some of the most valuable feedback that you can get to improve your course.  I've gotten a lot of bad feedback too, but I've gotten some really good feedback on smile sheets that has told me - you really need to improve that exercise.  Or, have more stuff going on in the afternoon that's fun because people were tired - stuff like that.  Okay, you hear that over and over - you better change it.

Thalheimer: Another thing I do with smile sheets is I also like to follow up two weeks later.  I follow up and I do a couple of interesting things.  One is I ask the same question on the same six point scale about how valuable you thought the training was.  Now they've gone back to the work place.  They now really know how valuable it is.  So it gives a better anchoring.

I’ve learned a couple of other things from looking at measurement from a learning fundamental standpoint.  One thing is, if we measure learning right at the end of learning, what is it we are measuring?  We're measuring the learning intervention’s ability to create understanding.  Which is fine, that's good, but we're not measuring the learning intervention’s ability to minimize forgetting.  You have a full day session or you have an e-learning program and you measure people right away - everything is top of mind.

T/D: You should be able to recall it.

Thalheimer: You can easily recall it and not only that, but you feel confident that you can recall it.  So that's a really biased way to get information.  That's one of the reasons that make it a bad proxy for your ability to remember it on the job.The final mistake we make is to measure something in the same context that people learned it.  That context is going to remind them of what they learned and because of that they're going to get better results than if we measured them in a different context.  So we need to do one of two things - we either need to know that our results are biased because we're in the same context that they learned in.  Or we need to change the context and make it more like the real world context so that it's more realistic.  It's more predictive of that real world environment.

T/D: Great points! Thank you Will.

Will Thalheimer founded Work-Learning Research which is a consulting practice that helps clients build more effective learning interventions.  He’s been in the learning and performance field since 1985.  His professional focus is on bridging the gap between the research side and the practice side