Best way to gauge comprehension of written training docs by employees?
April 17, 2013 8:09 AM Subscribe
Reading is important, and I want my team to do it as part of training. And then I want to know what they understand, how much they assimilate, what conclusions they reach from the articles and documents I am sending them. What are some good ways for me to accomplish that?
I am training a group of remote employees. We use real-time, live, interactive tools like chat, forum, google hangout, etc, but a large part of what needs to be done is reading (blogs and news articles, and training documentation).
I need a way to get feedback from them that accomplishes the following:
- Confirms that they read the material within the timeline I established.
- Encourages engagement, active reading, as a way to ensure better comprehension (as opposed to passive, mindless reading).
- Surfaces problem areas or individual challenges I should pay attention to.
For example, should I send a questionare? Ask for opinions on a certain passage? What are the best ways to engage them?
I don't mind if it resembles what your high school teacher did. It can look like school work, not a problem. If it helped you understand and analyze what you read, I want to try it!
I am training a group of remote employees. We use real-time, live, interactive tools like chat, forum, google hangout, etc, but a large part of what needs to be done is reading (blogs and news articles, and training documentation).
I need a way to get feedback from them that accomplishes the following:
- Confirms that they read the material within the timeline I established.
- Encourages engagement, active reading, as a way to ensure better comprehension (as opposed to passive, mindless reading).
- Surfaces problem areas or individual challenges I should pay attention to.
For example, should I send a questionare? Ask for opinions on a certain passage? What are the best ways to engage them?
I don't mind if it resembles what your high school teacher did. It can look like school work, not a problem. If it helped you understand and analyze what you read, I want to try it!
My experience, having set up and run remote teams (in my case, India) for five years:
There is no substitute spending actual, face to face time with them. I cannot repeat this more. I learnt this the hard way when I found my team getting even basic things wrong when after several remote training sessions and assurances that they understood how to do things. My remote team were smart and willing, we put comprehensive documentation together, but when we dug deeper we found that the actual implementation of what we wanted was being done in all sorts of bespoke ways we had not anticipated.
In some cases where we asked the team to do something new the remote team put their own documentation together. This was OK. But these process manuals were pages thick, meticulously put together, and often missed the point because what they actually did as opposed to what the process documentation suggested was being done were miles apart. The process manuals were an idealised view of how things should be done.
So:
- If your remote team come from a culture where acquiescence to one's bosses is the norm, then they will read the material, actively. But they may not raise problems or challenges. Or, as we found they were raising them internally within the group but not externally to us - it was perceived as a failure to not be able to get on and do the job. We also found that the team would sometimes say one thing and the very next moment go off and and do things how they wanted to.
- If your remote team are junior or inexperienced then they will struggle to synthesise the information in the way that you want and apply it how you want. If they are experienced this becomes less of an issue.
Is a questionnaire appropriate? Maybe, if the information is basic and quite binary (for example, my old company used to use online testing tools for things like data security and health and safety). If not then I don't think it is appropriate.
It is more appropriate, and long term will get you where you want more quickly to communicate regularly and clearly. To have regular catch up sessions or meetings, to create a culture of openness that allows your team to raise issues without fear they will look stupid. It is hard to do this remotely.
Your mileage may vary, but if you are training something complex, or to an inexperienced team, or are unsure of how the culture of the individuals, then spending more time training actively (i.e. not through just reading) and face to face will pay dividends.
Once you have reached a level of understanding of what issues the team are likely to have, and you know how well they can process information the way you want remotely then things like chat, forum, hangout etc work really well because they keep the channels open. But they are not a substitute for proper investment in training.
posted by MuffinMan at 8:31 AM on April 17, 2013 [1 favorite]
There is no substitute spending actual, face to face time with them. I cannot repeat this more. I learnt this the hard way when I found my team getting even basic things wrong when after several remote training sessions and assurances that they understood how to do things. My remote team were smart and willing, we put comprehensive documentation together, but when we dug deeper we found that the actual implementation of what we wanted was being done in all sorts of bespoke ways we had not anticipated.
In some cases where we asked the team to do something new the remote team put their own documentation together. This was OK. But these process manuals were pages thick, meticulously put together, and often missed the point because what they actually did as opposed to what the process documentation suggested was being done were miles apart. The process manuals were an idealised view of how things should be done.
So:
- If your remote team come from a culture where acquiescence to one's bosses is the norm, then they will read the material, actively. But they may not raise problems or challenges. Or, as we found they were raising them internally within the group but not externally to us - it was perceived as a failure to not be able to get on and do the job. We also found that the team would sometimes say one thing and the very next moment go off and and do things how they wanted to.
- If your remote team are junior or inexperienced then they will struggle to synthesise the information in the way that you want and apply it how you want. If they are experienced this becomes less of an issue.
Is a questionnaire appropriate? Maybe, if the information is basic and quite binary (for example, my old company used to use online testing tools for things like data security and health and safety). If not then I don't think it is appropriate.
It is more appropriate, and long term will get you where you want more quickly to communicate regularly and clearly. To have regular catch up sessions or meetings, to create a culture of openness that allows your team to raise issues without fear they will look stupid. It is hard to do this remotely.
Your mileage may vary, but if you are training something complex, or to an inexperienced team, or are unsure of how the culture of the individuals, then spending more time training actively (i.e. not through just reading) and face to face will pay dividends.
Once you have reached a level of understanding of what issues the team are likely to have, and you know how well they can process information the way you want remotely then things like chat, forum, hangout etc work really well because they keep the channels open. But they are not a substitute for proper investment in training.
posted by MuffinMan at 8:31 AM on April 17, 2013 [1 favorite]
Assign everyone a specific set of materials that they will then need to "teach" to the rest of the team during a live webcam session. Have the other team members evaluate their peer on depth of understanding of material, clarity in presenting it to others, completeness of material assigned, and to find and list specific mistakes or omissions.
This requires the entire team to actively read the material so they can properly evaluate the presenter. Of course you will be there to jump in at the end to correct mistakes, provide detail on anything left out, and answer questions from everyone.
Make it clear from the start that feedback such as "it was great; no problems noted" won't be acceptable. Encourage the presenter to ask specific questions of each team member throughout the presentation to prove that everyone did the reading and understands it (i.e., Tom, based on this new training data, how would you respond to a customer who asks XYZ?)
posted by trivia genius at 8:50 AM on April 17, 2013 [1 favorite]
This requires the entire team to actively read the material so they can properly evaluate the presenter. Of course you will be there to jump in at the end to correct mistakes, provide detail on anything left out, and answer questions from everyone.
Make it clear from the start that feedback such as "it was great; no problems noted" won't be acceptable. Encourage the presenter to ask specific questions of each team member throughout the presentation to prove that everyone did the reading and understands it (i.e., Tom, based on this new training data, how would you respond to a customer who asks XYZ?)
posted by trivia genius at 8:50 AM on April 17, 2013 [1 favorite]
I am an instructional designer, but I am not your instructional designer.
First, you need to set out very clear objectives that will help measure learning. You can use Bloom's taxonomy to author said objectives. The great thing is that the verbs you select pretty much drive how you're going to structure the training and assess how much knowledge they've retained.
Next, you'll want to make sure that the content is chunked and sequenced appropriately so that the information presented is directly related to a specific objective. It's also important to consider alternative delivery methods; there are different learning styles, and your learners may not be getting the support they need.
Testing, especially for languages, is a whole 'nother kettle of fish. LINCS have an excellent summary of assessment strategies for reading, and the section on silent reading comprehension will have lots of useful information, especially with regards to the different approaches you can take.
Feel free to PM me if you have any questions!
posted by evoque at 8:59 AM on April 17, 2013 [6 favorites]
First, you need to set out very clear objectives that will help measure learning. You can use Bloom's taxonomy to author said objectives. The great thing is that the verbs you select pretty much drive how you're going to structure the training and assess how much knowledge they've retained.
Next, you'll want to make sure that the content is chunked and sequenced appropriately so that the information presented is directly related to a specific objective. It's also important to consider alternative delivery methods; there are different learning styles, and your learners may not be getting the support they need.
Testing, especially for languages, is a whole 'nother kettle of fish. LINCS have an excellent summary of assessment strategies for reading, and the section on silent reading comprehension will have lots of useful information, especially with regards to the different approaches you can take.
Feel free to PM me if you have any questions!
posted by evoque at 8:59 AM on April 17, 2013 [6 favorites]
evoque's link to Bloom's taxonomy will definitely be useful for you if you're not already familiar with it.
I teach adults in a college setting. When I assign readings, I have them follow a series of steps that are meant to help them build understanding while allowing me to check their progress. The first two steps are done on their own; the next two are done together with their classmates.
posted by hurdy gurdy girl at 11:21 AM on April 17, 2013
I teach adults in a college setting. When I assign readings, I have them follow a series of steps that are meant to help them build understanding while allowing me to check their progress. The first two steps are done on their own; the next two are done together with their classmates.
- Students (in your case, trainees) read the material on their own. If they don't do this part, they won't be able to do the rest of the steps, so letting them know what's expected of them in steps 2, 3, and 4 will be good motivation for getting the reading done.
- After reading, they write a short response (1 page or so) to what they've read. This is intended to be informal and personal (i.e. you're not checking for grammar, but for ideas). I give them a list of suggestions and prompts, e.g. what's the main point of the article? What did it remind me of in my own experience or that I've read or heard about elsewhere? I also encourage them to write down any questions they had about the text. When you read over these responses, you will get a better sense of what they understand and what they're struggling with.
- After writing their response, students formulate two discussion questions and bring them to the large class meeting. Students often learn very well from discussing things with each other, and if they've done steps 1 and 2, they'll be prepared for this step. Be sure to provide examples of the kinds of discussion questions you want (e.g. not answerable with yes/no) before they do this step. Students get in small groups and take turns sharing their discussion questions with each other. The group takes notes on the discussion, then each small group reports back to the whole group. I don't know how easy this will be in a distance learning environment, but ideally you'd find a way to "listen in" and circulate among the groups while they're discussing.
- We repeat steps 1-3 with a series of related readings (3-5), and then at the end of the unit I provide a selection of essay prompts designed to show how they have synthesized the information in the readings. Students write a short essay exam in class on the prompt of their choice.
posted by hurdy gurdy girl at 11:21 AM on April 17, 2013
What's the military training slogan?
"See one, do one, teach one."
Or something like that and they know a few things about spreading institutional knowledge.
posted by trinity8-director at 12:47 PM on April 17, 2013
"See one, do one, teach one."
Or something like that and they know a few things about spreading institutional knowledge.
posted by trinity8-director at 12:47 PM on April 17, 2013
This thread is closed to new comments.
posted by purenitrous at 8:20 AM on April 17, 2013 [1 favorite]