Is it a Knowledge Check or a Quiz?

a-plus

In the midst of designing a facilitator-led curriculum for a client, we were met with a conundrum: according to our SME(s), one particular class just had to have a quiz at the end.

There were many problems with this idea, including the fact that  none of the other 6 courses in the curriculum ended with a quiz and that the audience was new-hires - so how intimidating would a quiz be?

We finally compromised on a Knowledge Check - that way our SME felt fulfilled (and  we fulfilled compliance requirements) but the learners wouldn't be too intimidated  (we hoped).

What's the difference?

A quiz is used to check for comprehension. Did your attendees learn what you taught? A quiz can come in many forms - you might ask your learners to recognize an answer,  as in the case of a multiple choice text. You might ask them to recall an answer,  as in the case of fill-in-the-blank. Or you may ask them to think of the answer  by giving a "case" and asking: What should you do next? In all cases the results  of the test matter. There is a score (perhaps numeric, perhaps pass/fail). There is a record of that score. And often the scores are compared to one another - resulting in a ranking of some sort.

Alternatively, a knowledge check is more of a review. It's used to determine if  the learners can find the answer. They are often allowed to use their learning materials  (handouts, workbooks, etc.) and potentially to work together. A knowledge check  might be in the form of a game (such as jeopardy) or it might be a solitary activity.  Knowledge checks are often used to help solidify the learning, allow learners to review the content one more time, and enable them to leave the training more confident in what they learned.

A knowledge check is appropriate in all situations; a quiz is only appropriate if  you have to ensure people know the answers before they leave training. There is  some consequence to not knowing the answers (such as performing the job incorrectly),  and you need to prove the "results" of the training.

Interview with Will Thalheimer, PhD

Thalheimer (1)

What motivated you to write this book?

I've worried about my own smile sheets (aka response forms, reaction forms, level 1's) for years! I know they're not completely worthless because I got useful feedback when I was a mediocre leadership trainer-feedback that helped me get better.

But I've also seen the research (two meta-analyses covering over 150 scientific studies) showing that smile sheets are NOT correlated with learning results-that is, smile sheets don't tell us anything about learning! I also saw clients-chief learning officers and other learning executives-completely paralyzed by their organizations' smile-sheet results. They knew their training was largely ineffective, but they couldn't get any impetus for change because the smile-sheet results seemed fine. 

So I asked myself, should we throw out our smile sheets or is it possible to improve them? I concluded that organizations would use smile sheets anyway, so we had to try to improve them. I wrote the book after figuring out how smile sheets could be improved.

If you could distill your message down to just one - what would it be?

Smile sheets should (1) draw from the wisdom distilled from the science-of-learning findings, and (2) smile-sheet questions ought to be designed to (2a) support learners in making more precise smile-sheet decisions and (2b) should produce results that are clear and actionable. Too often we use smile sheets to produce a singular score for our courses. "My course is a 4.1!" But these sorts of numerical averages leave everyone wondering what to do. 

How can trainers use this book to assist them in the work that they do?

Organizations, and learning-and-development professionals in particular, can use my book to gain wisdom about the limitations of their current evaluation approaches. They can review almost 30 candidate questions to consider utilizing in their own smile sheets. They can learn how to persuade others in using this radical new approach to smile-sheet design. Finally, they can use the book to give them the confidence and impetus to finally make improvements in their smile-sheet designs-improvements that will enable them to create a virtuous cycle of continuous improvement in terms of their learning designs.

Getting valid feedback is the key to any improvement. My book is designed to help organizations get better feedback on their learning results. 

Do you have a personal motto that you live by?

Be open to improvement. Look for the best sources of information-look to scientific research in particular to enable practical improvements. Be careful. Don't take the research at face value. Instead, understand it in relation to other research sources and, most importantly, utilize the research from a practical perspective. 

Will Thalheimer, PhD, PresidentWork-Learning Research, Inc.

How To Assess Real Results From Your Corporate Training

The four levels of corporate training evaluation (and the futility of most training evaluation)  was discussed in this earlier blog post; but in this post we will discuss the types of training evaluation that allow you to assess real results.

Level three evaluations are the most logical evaluations to deploy because they get at the purpose of the training – to change people’s behavior on the job.  A Level three evaluation then determines if people have actually changed their behavior by either observing them in action, asking them for their own assessment, or asking for a third-party’s assessment.

Level three evaluations incorporate Level two evaluations because the evaluator is able to determine if the trainee is utilizing the knowledge that they acquired during the training and applying it to their work.

Level Three Evaluations

Observation – by a manager, quality control or even a training person An observation form must be utilized so that the evaluation is not subjective (Did the trainee acquire the customer information, using the five prescribed questions, in the correct order? vs. Did the trainee begin the customer interaction correctly?)

Personal assessment – is frequently used for Level three evaluations because many organizations find observation to be cumbersome (it requires asking a third-party to conduct it, it requires disseminating and retrieving information, and other administrative tasks which are all subject to not being completed). In a personal assessment the trainee, once they are back on the job for a period of tame (three weeks, three months) reports on their own changed behavior.

Questions utilized include:

• Have you applied the ___ process in  your day-to-day work?

• How many times a day would you say you utilize the process?

• Have you seen positive results from utilize the process?

• Can you provide an example of when you used the process and what the outcome was? 

These types of questions not only help the training department to understand how the training is being utilized on the job, they also cause the employee to realize how they have changed their behavior as a result of training, and further, if the individual has not changed their behavior, these types of assessments help to reinforce the fact the training is an investment the organization has made in that individual and it is an investment the organization intends to follow up on.

Level Four Evaluations

Level four evaluations then tell us whether the investment in the training was worth it. For example: if the intention was to increase sales, did sales numbers go up? These types of evaluation require a lot of number crunching AND require a baseline of data to compare against, which many organizations simply don’t possess.

Factors and Nuances

One nuance which makes Level four evaluations difficult to conduct is determining how long it will take for the training to become “the way we work.”  When can the training department be confident that what was taught is truly ingrained in to the trainee’s everyday work responsibilities?  In other words – when should the measurement take place? If a goal was set prior to the training process - say, increasing sales by 50%, and sales increase by only 20% in the first three months following training – would that be considered a failure? What if, instead, the trainees were able to increase their sales by 20% every quarter following the training? Then that outcome would far exceed the 50% goal.  So when is the “line in the sand” drawn and success or failure determined?

Another nuance is that the long-term effects of training can be quite difficult to factor. For instance, if the intent was to increase sales, the training department might evaluate the sales numbers three months or six months after the training; but rarely will they evaluate it again a year after the training. And in some cases, where sales results are residual, the ongoing effect of the training is never quantified. For instance, in insurance sales, teaching salespeople to cross-sell (e.g. selling an umbrella policy to a current homeowner’ policy owner) not only results in an immediate uptick in sales, but also, when the policy is renewed, that sales training results in an ongoing increase in sales.

Sadly, most companies don’t take the time to extrapolate their training outcomes to Level three and Level four. It is acknowledged that evaluation at these levels can be time consuming and cumbersome, but these results are crucial for training departments to measure and communicate their worth to the organization as a whole.

Training Evaluation - What Does It Tell Us? Not Much!

4 levels

Most companies who do conduct evaluation of their training programs will stop at Level 2 evaluations (see graphic).

Level one evaluations are often called smile-sheets or butts-in-seats evaluations. They are realistically opinion gauges. they ask too many questions, including questions about the facilitator’s knowledge and skill, the quality of the learning materials, the comfort of the training room or delivery methodology (e.g. if it were e-Learning), etc. Unfortunately, the responses provide little useable information in return. Smile-sheets could be revitalized and used to a better purpose with just a bit of tweaking of the questioning process.

Level two evaluations are intended to test knowledge. They are typically a type of test – either paper-and-pencil (or these days, computer generated) or a demonstration / performance of skill (for instance, if you are teaching an individual to run a cash register, you wouldn’t want to stop at simply asking them questions about cash register operations – you would want to see them physically operate the cash register as well).The biggest drawback of Level two evaluations is that they realistically gauge short-term memory. They are typically distributed immediately after the training concludes, so most individuals have a relatively good chance of passing that type of evaluation.

Level three and Level four evaluations - those that assess whether the training is being used on the job and whether the intended business impact of the training was realized, are more complicated to design and administer and more often than not, simply not utilized in most businesses.

If you’d like to learn more about effective training evaluation, see this associated posted: How to Assess Real Results From Your Corporate Training.

Interview with Learning Expert: Will Thalheimer

Measuring Learning Results

T/D: You've written a research in practice report called Measuring Learning Results which is an understanding of learning and focusing on measurement.  Can you tell us more about that?

Thalheimer: Over the years I've been immersed in learning and looking at the research on learning.  One of the things I noticed in our field is that there are some leverage points - things that affect the whole field, things that drive us to do what we do.  One of those is measurement.  How we measure affects how we practice instructional design.  So I'll share with you a couple of things that I found.  One thing is that smile sheets aren't good enough.

T/D: Right, don't we know that at a gut level?

Thalheimer: We may know it at a gut level, but we keep doing it. I did some research with the e-learning Guild and we looked at how it's the most popular method, after completion rate, for an e-learning program.  So they've done like a research meta-analysis on this, and they found that smile sheet ratings don't even correlate with learning results.

T/D: Interesting.

Thalheimer: They correlated like 0.09 - which is…

T/D: Negligible.

Thalheimer: Negligible.  Smile sheets don't correlate with on the job performance either.

T/D: What does your opinion have to do with your performance?  Nothing.

Thalheimer: Right, so that's intriguing.  So one of the things we're trying to do when we measure stuff is to get better.  There are really three reasons that you might want to measure learning - one is to support learning, to give the learners feedback.  The other thing is to improve the learning interventions we create.  So improve our learning designs.  The third thing is to prove our results to our organization; our certification, grading, that kind of thing.  So three main reasons there.  If we're going to improve our learning designs one of the things that we need to do is make sure that our learning metrics are predictive of people's actual performance.  If our smile sheets aren’t predictive, then they're no good at doing that, they're no good at giving us good information.

I've been playing with trying to improve my own smile sheet so I'm going to recommend a couple of things.  One - let's not ask people just overall questions.  Let's put in some real learning points that we wanted to get across, because people aren't very good at remembering everything after a session, right?  So if you ask them in general did you like it you're going to get general responses. Instead; this is one of the learning points that we talked about, how valuable was that to you?

Then I ask them - what's the likelihood that you're going to use this within the next two weeks?  I have from a zero to 100% in 10% increments.

T/D: That's brilliant.

Thalheimer: I also ask them - what's the probability in the next two weeks that you'll share this with one of your co-workers?  I also really focus on the open ended comments.  I find that those are really some of the most valuable feedback that you can get to improve your course.  I've gotten a lot of bad feedback too, but I've gotten some really good feedback on smile sheets that has told me - you really need to improve that exercise.  Or, have more stuff going on in the afternoon that's fun because people were tired - stuff like that.  Okay, you hear that over and over - you better change it.

Thalheimer: Another thing I do with smile sheets is I also like to follow up two weeks later.  I follow up and I do a couple of interesting things.  One is I ask the same question on the same six point scale about how valuable you thought the training was.  Now they've gone back to the work place.  They now really know how valuable it is.  So it gives a better anchoring.

I’ve learned a couple of other things from looking at measurement from a learning fundamental standpoint.  One thing is, if we measure learning right at the end of learning, what is it we are measuring?  We're measuring the learning intervention’s ability to create understanding.  Which is fine, that's good, but we're not measuring the learning intervention’s ability to minimize forgetting.  You have a full day session or you have an e-learning program and you measure people right away - everything is top of mind.

T/D: You should be able to recall it.

Thalheimer: You can easily recall it and not only that, but you feel confident that you can recall it.  So that's a really biased way to get information.  That's one of the reasons that make it a bad proxy for your ability to remember it on the job.The final mistake we make is to measure something in the same context that people learned it.  That context is going to remind them of what they learned and because of that they're going to get better results than if we measured them in a different context.  So we need to do one of two things - we either need to know that our results are biased because we're in the same context that they learned in.  Or we need to change the context and make it more like the real world context so that it's more realistic.  It's more predictive of that real world environment.

T/D: Great points! Thank you Will.

Will Thalheimer founded Work-Learning Research which is a consulting practice that helps clients build more effective learning interventions.  He’s been in the learning and performance field since 1985.  His professional focus is on bridging the gap between the research side and the practice side

5 General Rules for Workplace Tests Part-5

Rule #5

Ordering Questions

You may wish to group evaluation questions by topic or you may mix them up. Back on the job, the work people encounter won't show up in any kind of logical sequence – so mixing questions up has its merits.

On the other hand, keeping questions grouped allows you to easily spot if a learner just "didn't get" a particular topic. If all the questions on a similar topic are grouped together and the learner answers all or almost all of them wrong, either he needs retraining or the topic itself was poorly presented. If your same-topic-area questions are interspersed throughout the exam, it might be harder for you to spot a problem.

5 General Rules for Workplace Tests Part-4

Rule #4

Use Key Words

Key words assist the test-taker in figuring out the answer.

• Who triggers the respondent to look for a person or position

• What triggers the test taker to look for a thing or a process

• Where triggers them to look for a place or location

• When triggers them to look for a date or a period of time

• Why will signal them to look for reasoning.

5 General Rules for Workplace Tests Part-3

Rule #3

Stick to the Facts

Do not include trivial information - the only intention of which is to confuse the test taker. For instance: Bob and Ed left their office on K Street in Washington DC at 4:45 pm to travel to BWI airport for a 9:00 pm flight - how far is the airport from their office? The times given have nothing to do with the correct answer; in fact, Bob and Ed are irrelevant, too. 4

A better phrased question would be:

Using (a calculator, a map, an internet site) calculate the distance between K Street in downtown DC and the BWI airport.

5 General Rules for Workplace Tests Part-2

Rule #2

Give Adequate and Specific Instructions

Instructions are critical. Do everything you can to make sure test takers know WHAT to do, and WHEN and HOW to do it.

Examples:

• If there is a time requirement, state it. e.g. you must finish this section in 30 minutes

• If a tool or resource is allowed, state it. e.g. you may use a calculator for questions 11 – 20. The opposite is true as well - you may NOT use a calculator to complete this section. For each item in column A there is ONLY ONE correct answer in column B.

• It's also quite helpful to read the instructions out loud at the start of the test even when they are clearly written on the test itself. This will ensure that everyone hears, sees, and interprets the directions the same way and allows for questions before anyone begins.

5 General Rules for Workplace Tests Part-1

RULE #1

Do Not Trick Them

If you have not taught “it” in the training, it should not be on the test. In addition, your test questions should be stated in the same manner they were stated/taught in the class. For example: if you teach the three characteristics of steel, don't ask: Which one of these is NOT a characteristic of steel. It's hard for most people to have success with "null" answers and more importantly, why reinforce what you don’t want them to remember?

Delta Takes Training Evaluation to the 4th Degree

Delta flight operations training goes through level three evaluations for "everything we do," says Scott Nutter, Flight Operations General Manager, and level four evaluation by tying training data to operational performance and safety metrics.  

As a result, Delta has received several awards including Travel Weekly's Magellan Award and being named to Fortune's World's Most Admired Companies in 2011.

Delta and Northwest merged more than three years ago and had a smooth integration of their training initiatives by keeping focused on these important level 3 and level 4 outcomes.

March /  April 2012 Training Magazinep. 40