computer aided essay grading
October 7, 2010 4:44 PM Subscribe
Have you used software to help you grade student essays? Tell me about your experience.
I teach college freshman composition (among other things), and I tend to take *way* too much time grading students essays (I don't get them back quickly enough; it also absorbs time I need for class prep and other projects). So I'm just starting to consider the idea of using some sort of software to help me grade more quickly.
I'd like to know your opinion about paper-grading software. I'm not thinking of software that automatically scores student writing (a horrible and clearly unworkable idea imho), but rather software that would in theory help me streamline my grading process by to some degree automating the process of commenting on essays.
I'm used to sitting with a clipboard, a paper, and a cup of tea. And it it weren't for my habit of hemming and hawing and thinking digressively about each of my papers, this method would be pretty good--certainly more convenient that reading and grading on a laptop, making comments in the margins via Microsoft Word.
But I've noticed that there are some errors and problems that I end up pointing out repeatedly within the same set of papers, and I feel like I might be wasting time by writing out similar comments again and again (either as marginal notes, or in my end comments). I've therefore been imagining a program that would work with Microsoft Word to allow me to create menus of user-defined notes, for insertion into papers.
My hope would be not that I'd end up providing only boilerplate comments for my students, but that I'd paste-in (with some changes) comments about common or mechanical problems, giving me a little more room to say more about the argument and method of each student's paper, over and above the more standardized remarks.
Anyhow, in a fit of procrastination, I just Googled this subject, and came across a piece of software called T.A. Toolbar that looks pretty much like what I was envisioning (although I wish that it inserted marginal comments were into Word doc comment bubbles instead of in-line in boldface). Anyhow, they have some flash demos posted that might give a good sense of how TA Toolbar works with both marginal and end comments. I have to say that at first blush it looks pretty interesting.
So my questions are these:
1) Have you ever used software to assist in grading college essays?
2) What software have you used, and how useful was it?
3) How did it affect the quality of the feedback you provided to students?
4) What was the reaction of your department and colleagues to your use of this software?
5) Do you feel that students read and valued this feedback as much as handwritten remarks?
Finally, if you aren't a teacher but are or have been a university-level student, what would be your attitude towards the use of this sort of software in grading your written work in an introductory college composition class?
Many thanks for any replies.
I teach college freshman composition (among other things), and I tend to take *way* too much time grading students essays (I don't get them back quickly enough; it also absorbs time I need for class prep and other projects). So I'm just starting to consider the idea of using some sort of software to help me grade more quickly.
I'd like to know your opinion about paper-grading software. I'm not thinking of software that automatically scores student writing (a horrible and clearly unworkable idea imho), but rather software that would in theory help me streamline my grading process by to some degree automating the process of commenting on essays.
I'm used to sitting with a clipboard, a paper, and a cup of tea. And it it weren't for my habit of hemming and hawing and thinking digressively about each of my papers, this method would be pretty good--certainly more convenient that reading and grading on a laptop, making comments in the margins via Microsoft Word.
But I've noticed that there are some errors and problems that I end up pointing out repeatedly within the same set of papers, and I feel like I might be wasting time by writing out similar comments again and again (either as marginal notes, or in my end comments). I've therefore been imagining a program that would work with Microsoft Word to allow me to create menus of user-defined notes, for insertion into papers.
My hope would be not that I'd end up providing only boilerplate comments for my students, but that I'd paste-in (with some changes) comments about common or mechanical problems, giving me a little more room to say more about the argument and method of each student's paper, over and above the more standardized remarks.
Anyhow, in a fit of procrastination, I just Googled this subject, and came across a piece of software called T.A. Toolbar that looks pretty much like what I was envisioning (although I wish that it inserted marginal comments were into Word doc comment bubbles instead of in-line in boldface). Anyhow, they have some flash demos posted that might give a good sense of how TA Toolbar works with both marginal and end comments. I have to say that at first blush it looks pretty interesting.
So my questions are these:
1) Have you ever used software to assist in grading college essays?
2) What software have you used, and how useful was it?
3) How did it affect the quality of the feedback you provided to students?
4) What was the reaction of your department and colleagues to your use of this software?
5) Do you feel that students read and valued this feedback as much as handwritten remarks?
Finally, if you aren't a teacher but are or have been a university-level student, what would be your attitude towards the use of this sort of software in grading your written work in an introductory college composition class?
Many thanks for any replies.
Having taught comp, I have to say that I found this pretty inoffensive at first. The basic functions really seem no different than, say, giving the students a guide to common copy-editing marks and requiring that they learn to interpret those marks accordingly. It still entails detailed reading of student essays, for example.
But then I heard the phrase "a button for generic praise" and had a pretty strong negative reaction to that, both as a former student and an instructor. While it's one thing to automate pointing out, say, pronoun referents, there's something pretty skeevy about having a program generate generic phrases of praise.
Honestly, if it's taking you this long to grade, I'd suggest a few things. First, that you're giving them too much feedback. Honestly, students can't really absorb more than a few comments per page. When I graded, I would dispense a worksheet of proofreader's marks at the beginning of the semester. They would be required to learn these to comment on one another's papers (and usually I'd quiz them on it at the beginning of the semester). It really doesn't take long to mark up a paper for errors in this way once you've absorbed the rules for it. Then, I would limit myself to 1-2 brief written comments per page, and a small handful of sentences at the end of the paper. Having taught (and graded) in mentored situations, this seemed to be more effective at getting them to actually read and absorb my comments than the detailed notes I saw one professor utilize, or the extremely limited rubric I saw used by another.
Also, if you're marking off the same errors over and over again in a set of papers, it's probably time to take a step back and actually have a discussion with your students about it during class. Yes, even if it's some ridiculous grammar issue like run-ons. It stinks that most students haven't had grammar instruction by the time they reach college, but you'll be doing them a huge favor in actually addressing these issues in class with them directly. When I took even a few minutes out of the beginning of class to discuss wide-spread writing issues with my students, I found the rate of improved writing to be much, much better than if I just noted it on each student's paper.
(Also, it helped that I usually told them that I'd mark down more than I normally would for that issue in the future!)
posted by PhoBWanKenobi at 5:31 PM on October 7, 2010 [1 favorite]
But then I heard the phrase "a button for generic praise" and had a pretty strong negative reaction to that, both as a former student and an instructor. While it's one thing to automate pointing out, say, pronoun referents, there's something pretty skeevy about having a program generate generic phrases of praise.
Honestly, if it's taking you this long to grade, I'd suggest a few things. First, that you're giving them too much feedback. Honestly, students can't really absorb more than a few comments per page. When I graded, I would dispense a worksheet of proofreader's marks at the beginning of the semester. They would be required to learn these to comment on one another's papers (and usually I'd quiz them on it at the beginning of the semester). It really doesn't take long to mark up a paper for errors in this way once you've absorbed the rules for it. Then, I would limit myself to 1-2 brief written comments per page, and a small handful of sentences at the end of the paper. Having taught (and graded) in mentored situations, this seemed to be more effective at getting them to actually read and absorb my comments than the detailed notes I saw one professor utilize, or the extremely limited rubric I saw used by another.
Also, if you're marking off the same errors over and over again in a set of papers, it's probably time to take a step back and actually have a discussion with your students about it during class. Yes, even if it's some ridiculous grammar issue like run-ons. It stinks that most students haven't had grammar instruction by the time they reach college, but you'll be doing them a huge favor in actually addressing these issues in class with them directly. When I took even a few minutes out of the beginning of class to discuss wide-spread writing issues with my students, I found the rate of improved writing to be much, much better than if I just noted it on each student's paper.
(Also, it helped that I usually told them that I'd mark down more than I normally would for that issue in the future!)
posted by PhoBWanKenobi at 5:31 PM on October 7, 2010 [1 favorite]
Put out a detailed grading rubric, with all the points that you're looking for, and descriptors for your common errors. Then, circle what applies, give scores in those categories, and you're halfway there. Mark specifics places in text, give a light interpretive end-comment, and you're there.
Plus, you give the grading rubric out with the prompt, and that makes the students feel that not only do you have standards, but they are getting full disclosure of them at the outset.
Some comp professors of my acquaintance tend to grade mechanics on the first two pages, then have the students themselves learn to find the errors, mark the same errors on the remaining pages, then turn them back in with a memo showing that they "get it." It's a fine system, and works well either as a required revision or with some extra credit attached.
Errors you see repeatedly across a section are also best dealt with by marking, then lecturing with examples drawn from student work (with topics changed to protect the guilty).
posted by LucretiusJones at 6:00 PM on October 7, 2010 [2 favorites]
Plus, you give the grading rubric out with the prompt, and that makes the students feel that not only do you have standards, but they are getting full disclosure of them at the outset.
Some comp professors of my acquaintance tend to grade mechanics on the first two pages, then have the students themselves learn to find the errors, mark the same errors on the remaining pages, then turn them back in with a memo showing that they "get it." It's a fine system, and works well either as a required revision or with some extra credit attached.
Errors you see repeatedly across a section are also best dealt with by marking, then lecturing with examples drawn from student work (with topics changed to protect the guilty).
posted by LucretiusJones at 6:00 PM on October 7, 2010 [2 favorites]
I haven't used software for this, but I do always create a rubric with common faults and common praiseworthy features, and tick boxes on this for what applies. I think this is reassuring to students, because they see that they aren't the only one making certain errors.
Also I do all my comments on the computer, in a separate document for each student, and copy-paste relevant comments from one file to another.
I agree with PhoBWanKenobi about not having too many comments. I try to aim for one main comment about the most important feature of the writing that could be improved (i.e. the one thing that if they concentrate on it next time, it will make the biggest difference to their writing), and one or two pieces of (specific and targeted) praise. They'll only remember one or two things anyway. I always say if they want more feedback, they can come see me in my office hours.
After all, I see comments during grading as fulfilling two purposes: justifying the grade, and suggesting improvements for next time. The one or two specific comments suggest improvements, and the rubric mentioned above suffices for justification of their grade.
posted by lollusc at 6:14 PM on October 7, 2010 [1 favorite]
Also I do all my comments on the computer, in a separate document for each student, and copy-paste relevant comments from one file to another.
I agree with PhoBWanKenobi about not having too many comments. I try to aim for one main comment about the most important feature of the writing that could be improved (i.e. the one thing that if they concentrate on it next time, it will make the biggest difference to their writing), and one or two pieces of (specific and targeted) praise. They'll only remember one or two things anyway. I always say if they want more feedback, they can come see me in my office hours.
After all, I see comments during grading as fulfilling two purposes: justifying the grade, and suggesting improvements for next time. The one or two specific comments suggest improvements, and the rubric mentioned above suffices for justification of their grade.
posted by lollusc at 6:14 PM on October 7, 2010 [1 favorite]
This doesn't directly answer the question, but here goes:
I haven't used the kind of software you're asking about, although in grad school I knew other TAs who kept a file of standard comments for pasting into Word.
I, too, like to grade papers with a pencil and clipboard. The piece of "technology" that helped me streamline grading was a photocopied checklist/rubric sort of thing. I'd seen rubrics where each aspect of a paper got a numerical score and the numbers added up to the grade, but I never found them very satisfactory. Then on one of the college teaching blogs (I want to say ProfHack, but I can't find the entry now) I saw this idea for a checklist, and I adapted it for my own use. I think it accomplishes what you want to accomplish with the software, so I'll describe my process:
First, I brainstormed about a dozen gradeable aspects of paper-writing that seemed to really make the difference between good papers and not-so-good ones.
Then I made a table with two columns. On the left, I wrote down a description of what each aspect looks like when it's done well. On the right, I wrote a brief description of what it looks like when it's done poorly (with a few "OR"s thrown in since there are multiple ways of failing on each aspect).
No numbers. The checklist is for giving students information, not for calculating the grade.
At the end of the two-sided checklist, I added a large blank field labeled "The most important thing to work on in your next paper is:"
I handed out copies of the checklist to students before they wrote their papers, so that they could see what the expectations were.
After papers were turned in, I kept a copy of the checklist at hand while reading each paper and marked it up alongside the paper. The checklist markup was quite flexible: when evaluating each aspect, I could place a check mark in the relevant box (good/bad), jot notes ("mostly" in the good box, "see p. 3" in the bad box), circle/delete parts of the checklist descriptions to tailor them to the paper, or use numbers or symbols to key my marginal notes on the paper to the relevant boxes on the checklist. I would add written notes expanding on the checklist descriptions as needed--specifying how they applied to the case at hand--though after a semester or two of tweaking the checklist, I was surprised how accurately and adequately the succinct descriptions on the checklist covered what I was seeing in student papers.
In most cases, I would write my most expansive notes (still usually no more than a couple sentences) in the "what to work on in the next paper" section. It could contain praise as well as criticism; for example, "Build on the strength of your well-structured arguments by defining key terms more clearly." This was my favorite part of the checklist system--it turned the feedback information into actionable instructions. I kept copies of the commented checklists (easy to do if you stack them separately from the papers and run them through a photocopier before handing them back with the papers) and advised students that I would be checking subsequent papers to see whether they had improved on the specific aspects that I called out in the "what to work on" section.
If several students made the same factual or interpretational error with regard to the specific topic of the essay, and it required lengthier explanation, I could always type something up in Word and attach a copy to each paper.
With this method I got my typical grading time down to about 15-20 minutes per five-page paper. Special/difficult cases could still consume upwards of 45 minutes but with any luck there would only be a few of those per semester. I don't think the students liked the checklist as much as fully personalized comments, but nobody really objected, and I saw students improving on the areas that I asked them to work on.
---------
For general grading advice, I agree with the posters above: you get severely diminishing returns if you comment in depth on more than one or two flaws in a paper—most students just can't absorb that much information and make that many different adjustments in their writing from one paper to the next. This is why my checklist concludes by telling students what is "most important" to work on, not everything they could work on.
posted by Orinda at 6:42 PM on October 7, 2010 [2 favorites]
I haven't used the kind of software you're asking about, although in grad school I knew other TAs who kept a file of standard comments for pasting into Word.
I, too, like to grade papers with a pencil and clipboard. The piece of "technology" that helped me streamline grading was a photocopied checklist/rubric sort of thing. I'd seen rubrics where each aspect of a paper got a numerical score and the numbers added up to the grade, but I never found them very satisfactory. Then on one of the college teaching blogs (I want to say ProfHack, but I can't find the entry now) I saw this idea for a checklist, and I adapted it for my own use. I think it accomplishes what you want to accomplish with the software, so I'll describe my process:
First, I brainstormed about a dozen gradeable aspects of paper-writing that seemed to really make the difference between good papers and not-so-good ones.
Then I made a table with two columns. On the left, I wrote down a description of what each aspect looks like when it's done well. On the right, I wrote a brief description of what it looks like when it's done poorly (with a few "OR"s thrown in since there are multiple ways of failing on each aspect).
No numbers. The checklist is for giving students information, not for calculating the grade.
At the end of the two-sided checklist, I added a large blank field labeled "The most important thing to work on in your next paper is:"
I handed out copies of the checklist to students before they wrote their papers, so that they could see what the expectations were.
After papers were turned in, I kept a copy of the checklist at hand while reading each paper and marked it up alongside the paper. The checklist markup was quite flexible: when evaluating each aspect, I could place a check mark in the relevant box (good/bad), jot notes ("mostly" in the good box, "see p. 3" in the bad box), circle/delete parts of the checklist descriptions to tailor them to the paper, or use numbers or symbols to key my marginal notes on the paper to the relevant boxes on the checklist. I would add written notes expanding on the checklist descriptions as needed--specifying how they applied to the case at hand--though after a semester or two of tweaking the checklist, I was surprised how accurately and adequately the succinct descriptions on the checklist covered what I was seeing in student papers.
In most cases, I would write my most expansive notes (still usually no more than a couple sentences) in the "what to work on in the next paper" section. It could contain praise as well as criticism; for example, "Build on the strength of your well-structured arguments by defining key terms more clearly." This was my favorite part of the checklist system--it turned the feedback information into actionable instructions. I kept copies of the commented checklists (easy to do if you stack them separately from the papers and run them through a photocopier before handing them back with the papers) and advised students that I would be checking subsequent papers to see whether they had improved on the specific aspects that I called out in the "what to work on" section.
If several students made the same factual or interpretational error with regard to the specific topic of the essay, and it required lengthier explanation, I could always type something up in Word and attach a copy to each paper.
With this method I got my typical grading time down to about 15-20 minutes per five-page paper. Special/difficult cases could still consume upwards of 45 minutes but with any luck there would only be a few of those per semester. I don't think the students liked the checklist as much as fully personalized comments, but nobody really objected, and I saw students improving on the areas that I asked them to work on.
---------
For general grading advice, I agree with the posters above: you get severely diminishing returns if you comment in depth on more than one or two flaws in a paper—most students just can't absorb that much information and make that many different adjustments in their writing from one paper to the next. This is why my checklist concludes by telling students what is "most important" to work on, not everything they could work on.
posted by Orinda at 6:42 PM on October 7, 2010 [2 favorites]
I can't praise the rubric method highly enough. It takes a lot of time at the outset to make, but every hour you put into the rubric saves you well over an hour on the grading end of things. Plus, there are the benefits that lollusc describes. Attach a copy to each student's essay, or have them turn one in with their paper, and grading can be a breeze.
A method for correcting grammatical and spelling errors that I recently read about but haven't used yet is simply putting a check mark on the line where an error occurs. Don't tell them what the error, just put one check mark per error on the line. Then, before you put the final grade on the paper, hand back the essays and give students 10-15 minutes in class to go over their papers and find/correct the errors they're able to. When they hand their papers back to you the second time, all you have to do is go back through and comment on the patterns of errors that still exist. It sounds like that will take more time on your end than save it, but the article suggested that students are usually capable of correcting more than 60% of their own errors. And students who have fewer errors will be able to leave class early, encouraging them to spend time correcting their errors before they turn the papers in for the first time, so over time the amount of work should go down. Plus, this method encourages students to actively learn, rather than passively receiving information about correct grammatical rules. I can dig up the article if you'd like; it's buried somewhere in the dark recesses of my hard drive.
As a student, I would be very frustrated if I learned my professor used a series of canned comments to respond to my paper. I have no problem with typed comments (as handwriting sometimes makes them illegible anyway), but if I knew my teacher was only interacting with my paper by clicking "insert comment #13" I would be royally pissed. It doesn't matter if most of the comments were specifically written for me; if I learned even a few were prewritten all the comments would be suspect and would therefore mean nothing to me. The professor would lose all credibility with me. But I'm one of those students who takes learning and teaching very seriously, so I'm most likely on the extreme end of the spectrum.
posted by lilac girl at 6:47 PM on October 7, 2010
A method for correcting grammatical and spelling errors that I recently read about but haven't used yet is simply putting a check mark on the line where an error occurs. Don't tell them what the error, just put one check mark per error on the line. Then, before you put the final grade on the paper, hand back the essays and give students 10-15 minutes in class to go over their papers and find/correct the errors they're able to. When they hand their papers back to you the second time, all you have to do is go back through and comment on the patterns of errors that still exist. It sounds like that will take more time on your end than save it, but the article suggested that students are usually capable of correcting more than 60% of their own errors. And students who have fewer errors will be able to leave class early, encouraging them to spend time correcting their errors before they turn the papers in for the first time, so over time the amount of work should go down. Plus, this method encourages students to actively learn, rather than passively receiving information about correct grammatical rules. I can dig up the article if you'd like; it's buried somewhere in the dark recesses of my hard drive.
As a student, I would be very frustrated if I learned my professor used a series of canned comments to respond to my paper. I have no problem with typed comments (as handwriting sometimes makes them illegible anyway), but if I knew my teacher was only interacting with my paper by clicking "insert comment #13" I would be royally pissed. It doesn't matter if most of the comments were specifically written for me; if I learned even a few were prewritten all the comments would be suspect and would therefore mean nothing to me. The professor would lose all credibility with me. But I'm one of those students who takes learning and teaching very seriously, so I'm most likely on the extreme end of the spectrum.
posted by lilac girl at 6:47 PM on October 7, 2010
Seconding the suggestion of making a handout with your top 20 comments and putting numbers in the margins corresponding to those. Even though it's very similar to the computer-pastes-prewritten-comments method, this way is more aboveboard about the prewritten-ness, which somehow makes it feel better.
An alternative is to limit yourself to one word comments: Expand. Omit. Wordy. (etc)
Good luck, I know grading sucks if you're conscientious about it.
posted by LobsterMitten at 7:37 PM on October 7, 2010
An alternative is to limit yourself to one word comments: Expand. Omit. Wordy. (etc)
Good luck, I know grading sucks if you're conscientious about it.
posted by LobsterMitten at 7:37 PM on October 7, 2010
For the class I TAed last semester, I had my students submit all assignments electronically, and I graded them on the computer. I hate typing comments though, so I still handwrote my comments in Word using the pen function with a tablet. This allowed me to scrawl freely and circle things like I would on paper, but if I had to make the same lengthy comments multiple times (e.g. correcting a common misconception) I was able to copy and paste my (handwritten) comment. These were sometimes mathy assignments, so this worked out great because I was able to handwrite the corrected equations and such. It's kind of a weird method, but it worked great for me. A bonus was that both the students and I would then retain a copy of the corrected assignments.
I've also used the rubric method, but I find it hard to restrain myself from writing out comments even with an elaborate numbered comment system. The ability to copy and paste handwritten comments worked best for my grading style.
posted by pemberkins at 7:40 PM on October 7, 2010
I've also used the rubric method, but I find it hard to restrain myself from writing out comments even with an elaborate numbered comment system. The ability to copy and paste handwritten comments worked best for my grading style.
posted by pemberkins at 7:40 PM on October 7, 2010
This thread is closed to new comments.
Also, I've had a few professors that year after year had people make the same mistakes that the handout was made and given at the start of the year.
As a student, it would kind of depend on what kind of grades I got. Good ones and I wouldn't mind. Bad ones and I'd maybe start to wonder if you actually read my paper. I also wouldn't put it past some people to learn how to trick your program into thinking they had a good paper.
posted by theichibun at 5:06 PM on October 7, 2010