Interview with Learning Expert: Will Thalheimer
Measuring Learning Results
T/D: You've written a research in practice report called Measuring Learning Results which is an understanding of learning and focusing on measurement. Can you tell us more about that?
Thalheimer: Over the years I've been immersed in learning and looking at the research on learning. One of the things I noticed in our field is that there are some leverage points - things that affect the whole field, things that drive us to do what we do. One of those is measurement. How we measure affects how we practice instructional design. So I'll share with you a couple of things that I found. One thing is that smile sheets aren't good enough.
T/D: Right, don't we know that at a gut level?
Thalheimer: We may know it at a gut level, but we keep doing it. I did some research with the e-learning Guild and we looked at how it's the most popular method, after completion rate, for an e-learning program. So they've done like a research meta-analysis on this, and they found that smile sheet ratings don't even correlate with learning results.
T/D: Interesting.
Thalheimer: They correlated like 0.09 - which is…
T/D: Negligible.
Thalheimer: Negligible. Smile sheets don't correlate with on the job performance either.
T/D: What does your opinion have to do with your performance? Nothing.
Thalheimer: Right, so that's intriguing. So one of the things we're trying to do when we measure stuff is to get better. There are really three reasons that you might want to measure learning - one is to support learning, to give the learners feedback. The other thing is to improve the learning interventions we create. So improve our learning designs. The third thing is to prove our results to our organization; our certification, grading, that kind of thing. So three main reasons there. If we're going to improve our learning designs one of the things that we need to do is make sure that our learning metrics are predictive of people's actual performance. If our smile sheets aren’t predictive, then they're no good at doing that, they're no good at giving us good information.
I've been playing with trying to improve my own smile sheet so I'm going to recommend a couple of things. One - let's not ask people just overall questions. Let's put in some real learning points that we wanted to get across, because people aren't very good at remembering everything after a session, right? So if you ask them in general did you like it you're going to get general responses. Instead; this is one of the learning points that we talked about, how valuable was that to you?
Then I ask them - what's the likelihood that you're going to use this within the next two weeks? I have from a zero to 100% in 10% increments.
T/D: That's brilliant.
Thalheimer: I also ask them - what's the probability in the next two weeks that you'll share this with one of your co-workers? I also really focus on the open ended comments. I find that those are really some of the most valuable feedback that you can get to improve your course. I've gotten a lot of bad feedback too, but I've gotten some really good feedback on smile sheets that has told me - you really need to improve that exercise. Or, have more stuff going on in the afternoon that's fun because people were tired - stuff like that. Okay, you hear that over and over - you better change it.
Thalheimer: Another thing I do with smile sheets is I also like to follow up two weeks later. I follow up and I do a couple of interesting things. One is I ask the same question on the same six point scale about how valuable you thought the training was. Now they've gone back to the work place. They now really know how valuable it is. So it gives a better anchoring.
I’ve learned a couple of other things from looking at measurement from a learning fundamental standpoint. One thing is, if we measure learning right at the end of learning, what is it we are measuring? We're measuring the learning intervention’s ability to create understanding. Which is fine, that's good, but we're not measuring the learning intervention’s ability to minimize forgetting. You have a full day session or you have an e-learning program and you measure people right away - everything is top of mind.
T/D: You should be able to recall it.
Thalheimer: You can easily recall it and not only that, but you feel confident that you can recall it. So that's a really biased way to get information. That's one of the reasons that make it a bad proxy for your ability to remember it on the job.The final mistake we make is to measure something in the same context that people learned it. That context is going to remind them of what they learned and because of that they're going to get better results than if we measured them in a different context. So we need to do one of two things - we either need to know that our results are biased because we're in the same context that they learned in. Or we need to change the context and make it more like the real world context so that it's more realistic. It's more predictive of that real world environment.
T/D: Great points! Thank you Will.
Will Thalheimer founded Work-Learning Research which is a consulting practice that helps clients build more effective learning interventions. He’s been in the learning and performance field since 1985. His professional focus is on bridging the gap between the research side and the practice side
Online Collaboration MUST be Designed
One of the most wonderful things about the online classroom is the ability to bring learners together who may otherwise be geographically separated. If one individual in New York and one individual in Arizona need the same training, the virtual classroom not only allows them to partake in that training without travel, but also to take that training with fellow learners.
Too often, however, the virtual classroom is used in presentation-mode rather than in collaborative-mode.
All virtual classroom platforms pledge that their product enables your organization and your learners to work collaboratively. And it is true. All virtual classroom platforms allow for learners to interact verbally, via chat or instant messenger, through the use of feedback symbols or emoticons, and often through breakout rooms which enable smaller discussions and group activities to occur.
This doesn't just happen spontaneously, however. It is imperative that the training be designed to be collaborative.
One of the basic tenets of adult learning is that adults prefer to learn collaboratively; in other words adults prefer to learn with others. Therefore, it is imperative that the focus of the learning process is on the learners working together, discussing, questioning, problem solving, and in general, contributing to the learning process and the learning content.
As Instructional Designers, we must put quite a bit of thought into how we can ensure the learners work together to achieve the learning outcome, rather than sitting at their individual sites being passive recipients of a presentation.
Basic Rules of Test Creation
With the recent popularity of LMS systems that call for testing and tracking of student results, our participants are starting to see a lot of quizzes, assessments, and certifications that are designed by a course designer or – worse – a subject matter expert.
Very few trainers (and no SME’s) have had training in how to create tests that are fair to the participant and legally defensible for the organization. Creating an equitable test is more than just asking 20 questions about the content. A test for knowledge is constructed quite differently than a test for skill. Tests for knowledge, such as paper and pencil tests, need to be constructed in such a way that the correct choice is unambiguous. They should not confuse or purposefully stump the test-taker.
Performance tests need to assess the participant’s skill as well as knowledge of the task or process. Often a person can prove they know what to do – but cannot actually perform the task.
Before inflicting an “exam” process on your participants, you must know what you are testing, why you are testing, and what you intend to do with the test results because the impact of testing is far reaching and can often be damaging to individuals as well as organizations.
Tech Industry Thinks to the Future
New York City's recently built Academy for Software Engineering (high school) is supported with direct help from many leaders in the tech industry, including Google, eBay, Facebook and FourSquare.
Keep It Suitably Simple
While there is still a need for formally-packaged courses, these are for special occasions, when we or our employers require some formal record of achievement (or at least of participation). In the meantime, there's a job to be done, and that's far better achieved through access to videos, PDFs, forums, blogs and simple web articles. These are much easier to produce than highly-structured e-learning and just as easy to consume. Nothing lasts more than five minutes and the emphasis is strictly on practical application.
Excerpted from Clive on Learning - Clive Shephard's blog. You can read the full text here: ttp://tinyurl.com/cc9kwhn
Bring Your Wii to Work!
Too often, e-learning modules end up being glorified PowerPoint presentations. The learner reads through the information in a linear, beginning-to-end format, and is tested for knowledge retention at the end. As detailed in 5 Gaming Elements for Effective E-Learning (Training Industry Quarterly, Fall 2012), there are five takeaways from video games that can take e-learning to the next level.
Contextualization takes the e-learning out of the void, and puts it into a time and place, such as a scenario or story that provides the back bone for the training.
Curiosity draws the learner into exploring the e-learning module, enticing them to completeness.
Control allows participants to direct their own learning, driving the direction of the training, and causing them to retain more information due to engagement.
Cooperation / Socialization integrates a very popular factor of many online games, removing feelings of isolation and fostering teamwork.
Engagement / Interactivity puts the learners in situations where they are participating in the training from the start, rather than at the end of a module.
And with the growing popularity of BYOD (bring your own device) we could have everyone bring their Wii controller to work! <grin>
Formalize Informal Training in Your Organization
About 80% of the training that occurs in the workplace doesn't occur in a formal training program. About 80% of the training that occurs is just one person assisting another in an informal way. You stand up and look over your cubicle and ask your cubicle mate, “Do you know how to take text out of table and just make it into a paragraph?” Or, a sales manager decides he's going to take his administrative assistant out on the road for a day so she can actually meet the customers and better understand what their customer's needs are.
This interview, with Dr. Nanette Miner, will discuss ways to formalize informal learning in your organization.
= = = = = = = = =
Miner: Most of new-hire training is what we consider “follow Joe around” training. This means that you hook a new person up with a more experienced person - follow Joe around and he'll show you how to do your job. Although this is efficient, there are many problems with this style of training. If you have more than one person who is “Joe,” in this case, the training can be different from individual to individual because every trainer is going to emphasize what they think is important or perhaps show shortcuts, or “their way” of doing things which may not be the prescribed way of doing things. So while it is efficient and it doesn't take a formal training process, in the end you can actually have some pretty poorly trained new hires.
One of the things you can do to keep that process in place while making it a little more formal is to create check lists of training so that you have some kind of assurance that everybody's getting the same training process. For instance, in the retail industry there's a lot of turnover. Organizations tend to hire clerks on an individual basis. If you had a new hire training checklist you could at least ensure that everybody was getting the same training on the cash register. For instance you’d show them how to ring a cash sale, how to ring a charge sale, how to run a coupon, how to process a refund – these are a the topics any new hire would need to know, but you could “formalize” the training by prescribing the order of learning from easiest (cash sale) to hardest (refund).
Another way to formalize the training would be to recruit individuals who are interested in training. Believe it or not, there are a lot of people in organizations who love to transmit their knowledge to others and would be happy to do it for free. Recruit those people to serve as mentors or coaches for everybody, not just new hires, but everybody. They can be the go-to person when a new process needs to be created or a process runs into a problem; this person can be the one who figures it out and then trains everyone else in the “new way.”
Another idea would be to make training the responsibility of everybody in the organization. Require everyone to take on new learning and then share it with others. What we often do, as individuals, is figure something out on our own and say, “Oh cool, now I know how to do that,” and we don't ever share it with anybody else. I remember reading about a software company that made it everybody's responsibility to take on new learning to the point where it was in their performance review every year. What did you learn this year and how did you disseminate it to the rest of the organization? So, the employee might run a lunch and learn or they might write something up in the company newsletter.The point is that everyone is learning all the time, and we should formalize a way to share that learning.
T/D: Thank you Dr. Miner, those are great tips in making it everyone's responsibility and sharing the knowledge. Next month we will finalize this interview by focusing on Accessing Employee Training through your local College or University.
= = = = = = =
Dr Nanette Miner has been an instructional designer for over two decades. She is President and Managing Consultant for The Training Doctor which specializes in working with subject matter experts to take the knowledge from their heads and design learning in such a way that others can adopt and implement the training immediately. She is also the author of The Accidental Trainer and co-author of Tailored Learning: Designing a Blend that Fits.
Training for the Taking
It's odd, really, for an instructional design firm, we are all-for not reinventing the wheel. If there is training already "out there" and it fufills your need - even if you need to augment it a bit - why not take advantage of someone else's hard (and brilliant) work?
Here are some websites we've sourced recently, where you can find training offerings from A (Audacity) to Z (Zbrush).
Educator.com - From physics to music theory, Educator bills itself as having the most comprehensive math and science content on the web. Pay monthly or annually for discount.
Edudemic.com - Lots of free resources and links to free resources. Geared toward teachers but their ideas and findings are really universal to anyone trying to be the best educator they can be.
Lynda.com - A video-based, on-demand, portal for software and business skills. Pay monthly or annually for discount. Free trial.
KhanAcademy.org - Their tag line says it all, "Learn almost anything for free." The topics are more academically inclined and so would be ideal for assisting your workers with basic skills such as mathematics or sciences. Also video based in a really engaging delivery format.
And for some interesting factoids that you can use as icebreakers, energizers or to amuse, check out the Smithsonian's new website: www.SeriouslyAmazing.com.
Quotable: Jessica Matthews
You are only done learning when you've decided that you're done learning.
Quotable: Jessica Mathews, CEO and co-founder of SOccket
Incentivize your training - a great model
On September 29 a concert was held in New York City's Central Park to bring awareness to worldwide issues such as disease, poverty and lack of drinking water.
The concert was free and 60,000 people attended BUT they had to earn the right to attend. First, they had to register at a website. Then, they had to earn points to get in to a lottery to be awarded the free tickets. They earned points by watching various videos on the issues above and / or then forwarding those important messages to their friends via Facebook or Twitter.
What a GREAT model for making your training viral! Especially when you are constrained by having to disseminate your learning through asynchronous methods (in other words, people will engage in the learning on their own time and schedule). Why not have prizes or awards for completing the training and certain tasks along the way?
If you'd like to read more about the concert, click here
Your "smart phone" will make you dumber
Have you ever obligingly followed your GPS even though you were pretty sure it was steering you wrong (pun intended)? Have you ever followed your GPS to a location and, shortly thereafter, when you had to return, you realized you needed the GPS to do it?
While having the technology to save us time and save us from mistakes is wonderful, it also "saves us" from having to think. The more we don't have to think, the less capable we become of it.
Here is a simple experiment: pretend you are teaching how to tell time, on a clock. to a 7 year old. We have become so used to digital displays of time - on our microwave, cable box, telephone and car dash - that it is a struggle to explain how the hands and the numbers on a dial indicate the time. And that is just one, very simple, example.
More and more in our professional journals we see articles about mobile technology. With every person (practically) in possession of a smart phone or tablet, the field of training is increasingly obsessed with ways to "push" information and answers to the learner, rather than teaching people how to think, investigate, reason or create an answer on their own.
Smart devices may save us time in the short term, but in the long run, they will hobble our learners' ability to actually learn.
How much do you really know about learning?
Parade Magazine recently had an informative back-to-school quiz focused on learning facts. Follow this link to see how much you really know. After you answer each question it will tell you the research behind the answer. We'll give you a few clues because they are important to training:
Testing doesn't simply measure what you know-it reinforces what you know, says psychologist Henry Roediger III, Ph.D., of Washington University in St. Louis. Every time you summon facts from memory, you strengthen your brain's hold on the material. (Editor's Note: this is why, at The Training Doctor, we always say that the quiz is just the final part of the learning process.)
Research on what's known as the "spacing effect" shows that we form stronger and more lasting memories by exposing ourselves to information over time. Repeated cycles of learning, consolidating, and then re-encountering material fix it firmly in our minds.
It's much more effective to "interweave" different types of problems - mixing them up so you learn how to quickly identify which approach is needed to solve each one. For example, a study of baseball batters found that when different types of pitches - fastballs, curveballs, sinkers-were mixed up unpredictably during practice, the players became more adept at scoring a hit.
Quotable: Peter Casebow
"Some would say you can't control or plan for something like informal learning, but you can put a strategy in place.Based on our experience, any strategy for informal learning needs to include three basic areas: improving basic skills, such as searching for information effectively, creating opportunities and encouraging sharing and collaboration."
Quotable: Peter Casebow, CEO of GoodPractice
Employee training leads to strong business performance
The #1 focus of Hubert Joly, Best Buy's new CEO, is employee training. He believes that inconsistent training across stores is what has led to uneven performance among stores and overall sales decline. His mission is to make store employees an "undisputed point of reference" for customers.
Source: Chicago Tribune
Quotable: Peter Casebow
"Some would say you can't control or plan for something like informal learning, but you can put a strategy in place. Based on our experience, any strategy for informal learning needs to include three basic areas: improving basic skills, such as searching for information effectively, creating opportunities and encouraging sharing and collaboration."
Quotable: Peter Casebow, CEO of GoodPractice
Create a web page for exclusively for your next training class
We've been hearing a lot lately about the need to use different modalities in our learning (aka blended learning), the need to optimize technology, and the need to bring social learning to corporate America. Well, now you have the chance to try all of those things, risk free.
At www.Weebly.com you can create a website and host it for free "in the cloud." It has an easy drag-and-drop user interface which is very intuitive and doesn't require any skill or knowledge about web design. You can add pictures, audio, video, documents (such as uploading your slide set after a classroom training, for instance) and even create a blog so that you can post assignments, reminders and coaching tips, and ask for your learner's feedback and response.
Truly amazing - considering this service is free - is that Weebly automatically creates a mobile-friendly version of your site so your learners can connect via their smart phones, as well.
Give it a try for your next course and send us your web page - we'd love to share it!
Why the 2-hour training class doesn't work
Research on what’s known as the “spacing effect” shows that we form stronger and more lasting memories by exposing ourselves to information over time. Repeated cycles of learning, consolidating, and then re-encountering material fix it firmly in our minds.
When does the learning occur?
Let’s preface this article with two assumptions:
Most people who design training are not schooled in how to design training, these days. They are more typically subject matter experts.
Most people who ARE schooled in training still are not schooled in adult learning theory.
Over the years, we have noticed a recurring fault: designers (and facilitators) of training often equate participating in an activity with actually learning from it. Not so. Here are two examples:
A course in sales training, that we designed for a client, required a breakout group activity. At the end of the activity, each group was to present back their outcomes. As each group finished their presentation the facilitator replied with “good dialogue, good dialogue” and then moved on to the next group. The facilitator spent NO time asking for further interpretation of their decisions or outcomes, he spent NO time comparing one group’s response to another, he never asked any other group to respond to or ask questions of the presenting group.
A training design that we reviewed for a client included a 10-minute, independent, web browsing activity in which learners were to research the “key selling features” of a particular product. Once the activity was over, the course moved on to the next topic. There was no discussion of what people had found. There was no compare and contrast. There was no knowing if you did the activity right or not!
So the question we pose is this: When does the learning occur?
Lucky for you, we’ll give you the answer as well: It occurs in the debrief. The debrief happens after the activity. Rarely, if EVER, do participants get the ah-ha moment from simply participating in an activity. Ninety-nine percent of the time it occurs during the debrief when they are asked to process, sort and supply their findings. When they hear what others, or other groups, have come up with and compare it to their own. When they are asked “so what does this mean for you, on the job?”
Simply designing or facilitating an activity is not sufficient. The REAL learning comes after the activity. Discussions with peers, working collaboratively or competitively, all help to ensure that learners actually process what has just occurred and gain meaning from it.
Ensure that you think that far ahead when you design for your learners.
Quotable: Dr. Roth Tartell
Clearly, much of what the leader needs to do to increase employee engagement levels can be shaped through learning.
Learning professionals have a responsibility to their organizations to ensure that perspectives and approaches critical to successful engagement are built in to curricula, incorporated into developmental plans, and then included in the talent discussions that shape the future leaders of the organization.
Quotable: Dr. Roth Tartel is Learning and Development Manager - North America for GE Capital Real Estate
What you can learn about eLearning - from Engineers
Since 1980, The Society for Manufacturing Engineers – Education Foundation, has awarded over $31 million in grants, scholarships, and awards to high schoolers pursuing a degree in science, technology, engineering and math, more than any other professional engineering society.
Their website is a marvel of interactivity and engagement. It’s a wonderful model for e-learning as the ‘learner’ can pursue multiple topics and to multiple-depths, through their own decision making process. www.manufacturingiscool.com