Interview with Learning Expert: Will Thalheimer

Measuring Learning Results

T/D: You've written a research in practice report called Measuring Learning Results which is an understanding of learning and focusing on measurement.  Can you tell us more about that?

Thalheimer: Over the years I've been immersed in learning and looking at the research on learning.  One of the things I noticed in our field is that there are some leverage points - things that affect the whole field, things that drive us to do what we do.  One of those is measurement.  How we measure affects how we practice instructional design.  So I'll share with you a couple of things that I found.  One thing is that smile sheets aren't good enough.

T/D: Right, don't we know that at a gut level?

Thalheimer: We may know it at a gut level, but we keep doing it. I did some research with the e-learning Guild and we looked at how it's the most popular method, after completion rate, for an e-learning program.  So they've done like a research meta-analysis on this, and they found that smile sheet ratings don't even correlate with learning results.

T/D: Interesting.

Thalheimer: They correlated like 0.09 - which is…

T/D: Negligible.

Thalheimer: Negligible.  Smile sheets don't correlate with on the job performance either.

T/D: What does your opinion have to do with your performance?  Nothing.

Thalheimer: Right, so that's intriguing.  So one of the things we're trying to do when we measure stuff is to get better.  There are really three reasons that you might want to measure learning - one is to support learning, to give the learners feedback.  The other thing is to improve the learning interventions we create.  So improve our learning designs.  The third thing is to prove our results to our organization; our certification, grading, that kind of thing.  So three main reasons there.  If we're going to improve our learning designs one of the things that we need to do is make sure that our learning metrics are predictive of people's actual performance.  If our smile sheets aren’t predictive, then they're no good at doing that, they're no good at giving us good information.

I've been playing with trying to improve my own smile sheet so I'm going to recommend a couple of things.  One - let's not ask people just overall questions.  Let's put in some real learning points that we wanted to get across, because people aren't very good at remembering everything after a session, right?  So if you ask them in general did you like it you're going to get general responses. Instead; this is one of the learning points that we talked about, how valuable was that to you?

Then I ask them - what's the likelihood that you're going to use this within the next two weeks?  I have from a zero to 100% in 10% increments.

T/D: That's brilliant.

Thalheimer: I also ask them - what's the probability in the next two weeks that you'll share this with one of your co-workers?  I also really focus on the open ended comments.  I find that those are really some of the most valuable feedback that you can get to improve your course.  I've gotten a lot of bad feedback too, but I've gotten some really good feedback on smile sheets that has told me - you really need to improve that exercise.  Or, have more stuff going on in the afternoon that's fun because people were tired - stuff like that.  Okay, you hear that over and over - you better change it.

Thalheimer: Another thing I do with smile sheets is I also like to follow up two weeks later.  I follow up and I do a couple of interesting things.  One is I ask the same question on the same six point scale about how valuable you thought the training was.  Now they've gone back to the work place.  They now really know how valuable it is.  So it gives a better anchoring.

I’ve learned a couple of other things from looking at measurement from a learning fundamental standpoint.  One thing is, if we measure learning right at the end of learning, what is it we are measuring?  We're measuring the learning intervention’s ability to create understanding.  Which is fine, that's good, but we're not measuring the learning intervention’s ability to minimize forgetting.  You have a full day session or you have an e-learning program and you measure people right away - everything is top of mind.

T/D: You should be able to recall it.

Thalheimer: You can easily recall it and not only that, but you feel confident that you can recall it.  So that's a really biased way to get information.  That's one of the reasons that make it a bad proxy for your ability to remember it on the job.The final mistake we make is to measure something in the same context that people learned it.  That context is going to remind them of what they learned and because of that they're going to get better results than if we measured them in a different context.  So we need to do one of two things - we either need to know that our results are biased because we're in the same context that they learned in.  Or we need to change the context and make it more like the real world context so that it's more realistic.  It's more predictive of that real world environment.

T/D: Great points! Thank you Will.

Will Thalheimer founded Work-Learning Research which is a consulting practice that helps clients build more effective learning interventions.  He’s been in the learning and performance field since 1985.  His professional focus is on bridging the gap between the research side and the practice side

5 General Rules for Workplace Tests Part-5

Rule #5

Ordering Questions

You may wish to group evaluation questions by topic or you may mix them up. Back on the job, the work people encounter won't show up in any kind of logical sequence – so mixing questions up has its merits.

On the other hand, keeping questions grouped allows you to easily spot if a learner just "didn't get" a particular topic. If all the questions on a similar topic are grouped together and the learner answers all or almost all of them wrong, either he needs retraining or the topic itself was poorly presented. If your same-topic-area questions are interspersed throughout the exam, it might be harder for you to spot a problem.

5 General Rules for Workplace Tests Part-4

Rule #4

Use Key Words

Key words assist the test-taker in figuring out the answer.

• Who triggers the respondent to look for a person or position

• What triggers the test taker to look for a thing or a process

• Where triggers them to look for a place or location

• When triggers them to look for a date or a period of time

• Why will signal them to look for reasoning.

5 General Rules for Workplace Tests Part-3

Rule #3

Stick to the Facts

Do not include trivial information - the only intention of which is to confuse the test taker. For instance: Bob and Ed left their office on K Street in Washington DC at 4:45 pm to travel to BWI airport for a 9:00 pm flight - how far is the airport from their office? The times given have nothing to do with the correct answer; in fact, Bob and Ed are irrelevant, too. 4

A better phrased question would be:

Using (a calculator, a map, an internet site) calculate the distance between K Street in downtown DC and the BWI airport.

5 General Rules for Workplace Tests Part-2

Rule #2

Give Adequate and Specific Instructions

Instructions are critical. Do everything you can to make sure test takers know WHAT to do, and WHEN and HOW to do it.

Examples:

• If there is a time requirement, state it. e.g. you must finish this section in 30 minutes

• If a tool or resource is allowed, state it. e.g. you may use a calculator for questions 11 – 20. The opposite is true as well - you may NOT use a calculator to complete this section. For each item in column A there is ONLY ONE correct answer in column B.

• It's also quite helpful to read the instructions out loud at the start of the test even when they are clearly written on the test itself. This will ensure that everyone hears, sees, and interprets the directions the same way and allows for questions before anyone begins.

5 General Rules for Workplace Tests Part-1

RULE #1

Do Not Trick Them

If you have not taught “it” in the training, it should not be on the test. In addition, your test questions should be stated in the same manner they were stated/taught in the class. For example: if you teach the three characteristics of steel, don't ask: Which one of these is NOT a characteristic of steel. It's hard for most people to have success with "null" answers and more importantly, why reinforce what you don’t want them to remember?

Basic Rules of Test Creation

With the recent popularity of LMS systems that call for testing and tracking of student results, our participants are starting to see a lot of quizzes, assessments, and certifications that are designed by a course designer or – worse – a subject matter expert.

Very few trainers (and no SME’s) have had training in how to create tests that are fair to the participant and legally defensible for the organization. Creating an equitable test is more than just asking 20 questions about the content. A test for knowledge is constructed quite differently than a test for skill. Tests for knowledge, such as paper and pencil tests, need to be constructed in such a way that the correct choice is unambiguous. They should not confuse or purposefully stump the test-taker.

Performance tests need to assess the participant’s skill as well as knowledge of the task or process. Often a person can prove they know what to do – but cannot actually perform the task.

Before inflicting an “exam” process on your participants, you must know what you are testing, why you are testing, and what you intend to do with the test results because the impact of testing is far reaching and can often be damaging to individuals as well as organizations.