Help me counterbalance 200 questionnaires!
January 8, 2012 9:05 AM Subscribe
How can I effectively (and efficiently) counterbalance the questions for 200 questionnaires?
As part of my honours thesis I am distributing 200 paper questionnaires to some intro psych students. There are around 8 questions on the questionnaire.
I need to counterbalance the questions to ensure that any effect that I find wasn't a result of the order of the questions.
My prof (who is kind of old school) gave me an old counter-balancing numbers table, a page of numbers that allows you to choose random numbers. He suggested I number the questions, and then go through each questionnaire individually, copying and pasting the questions so that all 200 questionnaires become counterbalanced. THis seems extremely time consuming and old fashioned! There has got to be a better way.
I am willing to pay for the software needed to do this electronically if it's available. The problem is that these questionnaires are going to be given in hard copy, not electronically, so I can't use survey monkey or any other online resource that would allow me to counterbalance that way. I need to produce 200, counterbalanced hard copies of this questionnaire.
Is there any better way to do this than by hand? Right now the questionnaire is in microsoft word, the questions are only a couple of sentences long each, and the response is a 5 point likert scale.
Thanks brainiacs.
As part of my honours thesis I am distributing 200 paper questionnaires to some intro psych students. There are around 8 questions on the questionnaire.
I need to counterbalance the questions to ensure that any effect that I find wasn't a result of the order of the questions.
My prof (who is kind of old school) gave me an old counter-balancing numbers table, a page of numbers that allows you to choose random numbers. He suggested I number the questions, and then go through each questionnaire individually, copying and pasting the questions so that all 200 questionnaires become counterbalanced. THis seems extremely time consuming and old fashioned! There has got to be a better way.
I am willing to pay for the software needed to do this electronically if it's available. The problem is that these questionnaires are going to be given in hard copy, not electronically, so I can't use survey monkey or any other online resource that would allow me to counterbalance that way. I need to produce 200, counterbalanced hard copies of this questionnaire.
Is there any better way to do this than by hand? Right now the questionnaire is in microsoft word, the questions are only a couple of sentences long each, and the response is a 5 point likert scale.
Thanks brainiacs.
I used to have to randomly reorder my sampling order each day and found excel the fastest way to change list order (based on what I had available).
Copy and paste them into excel, one question per row. Then generate 200 lists of numbers containing from 1-8 in random order (there are plenty of free websites for this, I used to use www.randomizer.org/). Paste them into the columns next to the questions and sort by each column to reorder your questions. For each new order either back into word or where ever or just print from excel. Excel has it's own random number generation function built in but it's not very good and not very random so I never bothered with it, but you could do it that way too.
posted by shelleycat at 9:14 AM on January 8, 2012
Copy and paste them into excel, one question per row. Then generate 200 lists of numbers containing from 1-8 in random order (there are plenty of free websites for this, I used to use www.randomizer.org/). Paste them into the columns next to the questions and sort by each column to reorder your questions. For each new order either back into word or where ever or just print from excel. Excel has it's own random number generation function built in but it's not very good and not very random so I never bothered with it, but you could do it that way too.
posted by shelleycat at 9:14 AM on January 8, 2012
You can also generate pseudo-random numbers in the statistical software package R, if your professor prefers that you use that.
posted by dfriedman at 9:16 AM on January 8, 2012 [1 favorite]
posted by dfriedman at 9:16 AM on January 8, 2012 [1 favorite]
Response by poster: I am saying I want to randomly order these questions so that the order is different in each respondents' quiz.
This is promising news that maybe I can accomplish this in Excel! I'm just not quite sure how I will get the likert scale into Excel, or how I will copy and paste from Excel back into word without the format getting screwed up?
I'm going to go play around in Excel now and see if I can figure it out! Thanks for the quick responses!
posted by whalebreath at 9:26 AM on January 8, 2012
This is promising news that maybe I can accomplish this in Excel! I'm just not quite sure how I will get the likert scale into Excel, or how I will copy and paste from Excel back into word without the format getting screwed up?
I'm going to go play around in Excel now and see if I can figure it out! Thanks for the quick responses!
posted by whalebreath at 9:26 AM on January 8, 2012
Response by poster: Dfriedman: Thanks, but the RANDBETWEEN function seems to just provide me with one random number (ex: 4). I'm looking for a way to shuffle the questions over and over. Maybe I'm misunderstanding how to use it?
Shelleycat: Your answer seems super helpful but I'm having some trouble with a couple of the steps.
I've generated the 200 lists of random numbers from 1-8 (using randomizer.org). I can't seem to paste them directly into the columns, however. For example, randomizer gives me '1, 3, 8, 6, 7, 4, 5', so when I copy and paste that into excel, all of those numbers along with their commas just go into one cell. Do I have to manually copy the numbers into a column?
I think once I get the numbers into the columns I can figure out how to sort by each column.
Thanks smarties.
posted by whalebreath at 9:46 AM on January 8, 2012
Shelleycat: Your answer seems super helpful but I'm having some trouble with a couple of the steps.
I've generated the 200 lists of random numbers from 1-8 (using randomizer.org). I can't seem to paste them directly into the columns, however. For example, randomizer gives me '1, 3, 8, 6, 7, 4, 5', so when I copy and paste that into excel, all of those numbers along with their commas just go into one cell. Do I have to manually copy the numbers into a column?
I think once I get the numbers into the columns I can figure out how to sort by each column.
Thanks smarties.
posted by whalebreath at 9:46 AM on January 8, 2012
Response by poster: Shelleycat: I was wrong. I have no idea how to sort by each column.
posted by whalebreath at 9:47 AM on January 8, 2012
posted by whalebreath at 9:47 AM on January 8, 2012
OK here's how I do it: one randomization, which I get in Excel (assign random number to each item then sort lowest to highest by that number). Then half the subjects get that order, the other half get it in reverse order (in Excel, sort highest to lowest).
The logic is this: whatever order effects that could happen due to item X coming before Y in the first ordering will be negated by Y coming before X in the second ordering. This will be true for all Xes and Ys.
You could go crazy and use a latin squares design to get a purer randomization (for each item, there's an equal probability of getting each other item just prior to it etc.) But unless you have an a priori reason to worry that order effects will lead to spurious between-condition effects, it's not worth worrying about.
My advice to every young scientist is this: whatever you do, make sure you understand exactly why you did it. Telling your supervisor "I did it this way because that's what you told me to do" is not the right answer.
posted by drmarcj at 9:48 AM on January 8, 2012 [1 favorite]
The logic is this: whatever order effects that could happen due to item X coming before Y in the first ordering will be negated by Y coming before X in the second ordering. This will be true for all Xes and Ys.
You could go crazy and use a latin squares design to get a purer randomization (for each item, there's an equal probability of getting each other item just prior to it etc.) But unless you have an a priori reason to worry that order effects will lead to spurious between-condition effects, it's not worth worrying about.
My advice to every young scientist is this: whatever you do, make sure you understand exactly why you did it. Telling your supervisor "I did it this way because that's what you told me to do" is not the right answer.
posted by drmarcj at 9:48 AM on January 8, 2012 [1 favorite]
If you're at a University and If they use the blackboard course management system, and If they use the Respondus tools for randomizing test questions, then that could be useful.
posted by theora55 at 10:11 AM on January 8, 2012
posted by theora55 at 10:11 AM on January 8, 2012
I saved the random lists output as a comma delineated text file then imported into excel, so that it populated the right cells automagically (used notebook to save as the comma delineated .txt file, opened in excel using the usual file-open menu command, then copy/pasted the numbers using paste special to transpose them into columns).
You may find a different website gives you the lists in a better format to start with though, I seem to recall that research randomiser changed the output format at one point and it became too annoying. dfriedman's randbetween function is probably way easier in general if you can get it to work right (I never could but I didn't really try that hard). Look around, there are lots of places to randomly order numbers.
However you do it you'll end up with lists of numbers, each with digits from 1-8 in whatever order. I used to actually just paste them in one at a time and reorder but putting all 200 in at once is probably faster given the annoyances of importing them. Then you just use the normal sort function up in the menu to sort everything by the numbers in one column. That 1 to 8 which was all messed up turns around to be in order, and all the other columns get changed as well. So the questions are now in the order listed by the numbers rather than the order you wrote them. You may need to select everything to make sure all the columns change order (so you're not just sorting that one column), and definitely make sure there are no gaps in the block of data. Once sorted, copy/paste your questions list, then move across to the next column and sort by that one. Its numbers go into order, dragging everything else around into the new order, and so it goes.
I definitely think there is room here for having blocks of orders so some people get the same tests, be it two blocks as drmarcj says or something else. Because you're only taking a smallish subsample of the possible orders available, you're not correcting to stop the orders appearing more than one, and because randomness is uneven by nature, your set of orders may have clumps or biases built into it. So it's probably a lot of extra work for not much gain to go for 200 different tests.
posted by shelleycat at 10:30 AM on January 8, 2012
You may find a different website gives you the lists in a better format to start with though, I seem to recall that research randomiser changed the output format at one point and it became too annoying. dfriedman's randbetween function is probably way easier in general if you can get it to work right (I never could but I didn't really try that hard). Look around, there are lots of places to randomly order numbers.
However you do it you'll end up with lists of numbers, each with digits from 1-8 in whatever order. I used to actually just paste them in one at a time and reorder but putting all 200 in at once is probably faster given the annoyances of importing them. Then you just use the normal sort function up in the menu to sort everything by the numbers in one column. That 1 to 8 which was all messed up turns around to be in order, and all the other columns get changed as well. So the questions are now in the order listed by the numbers rather than the order you wrote them. You may need to select everything to make sure all the columns change order (so you're not just sorting that one column), and definitely make sure there are no gaps in the block of data. Once sorted, copy/paste your questions list, then move across to the next column and sort by that one. Its numbers go into order, dragging everything else around into the new order, and so it goes.
I definitely think there is room here for having blocks of orders so some people get the same tests, be it two blocks as drmarcj says or something else. Because you're only taking a smallish subsample of the possible orders available, you're not correcting to stop the orders appearing more than one, and because randomness is uneven by nature, your set of orders may have clumps or biases built into it. So it's probably a lot of extra work for not much gain to go for 200 different tests.
posted by shelleycat at 10:30 AM on January 8, 2012
That should say notepad up there, not notebook. Whatever free text editor your computer has. I'd just copy/paste the lists in there, one on each line. It should save as a comma delineated .txt file without a problem then, but depends on if extra characters and cruft get in there (it's been a while since I did it).
posted by shelleycat at 10:38 AM on January 8, 2012
posted by shelleycat at 10:38 AM on January 8, 2012
Here is a python script that will randomly generate a order a list of numbers between 1 and 8.
### Start code###
import random
ask =["1", "2", "3", "4", "5", "6", "7", "8"]
for x in range(8):
a = range(1,9)
questions = {}
for i in ask:
randomnum = random.choice(a)
a.remove(randomnum) ## remove picked random number
questions[i] = randomnum
print randomnum
raw_input("Press enter to order questions again. ")
posted by Brent Parker at 11:02 AM on January 8, 2012
### Start code###
import random
ask =["1", "2", "3", "4", "5", "6", "7", "8"]
for x in range(8):
a = range(1,9)
questions = {}
for i in ask:
randomnum = random.choice(a)
a.remove(randomnum) ## remove picked random number
questions[i] = randomnum
print randomnum
raw_input("Press enter to order questions again. ")
posted by Brent Parker at 11:02 AM on January 8, 2012
Randomization will probably be fine, but if you want to do things professionally a restricted randomization is preferable (you can easily by bad luck end up with a very unbalanced output). Note that if you are interested in certain questions having replicates of given orders is more useful than exploring more of all possible orders. If you are interested in citations, any text book on experimental design will discuss this. As a practical matter, it may save you substantial time on data entry to just pick a small number of orders.
I would modify the above suggestions by writing a short script (perl, python, R, whatever) which writes to an output file latex corresponding to the 200 questionnaires. If you do not have an auto-stapling printer/copy machine, then putting it all into a single file will allow you to compile to pdf and print in a single command; otherwise you will want to compile and print inside the loop or write a separate script. Make sure that somewhere on the page (possibly a footnote) is the index number, just in case. Be sure to save the random number generator seed that you initialize at.
Ms. Vegetable tells me that one could accomplish this same idea with VBA inside of word rather than using latex, but not the specifics.
posted by a robot made out of meat at 11:12 AM on January 8, 2012
I would modify the above suggestions by writing a short script (perl, python, R, whatever) which writes to an output file latex corresponding to the 200 questionnaires. If you do not have an auto-stapling printer/copy machine, then putting it all into a single file will allow you to compile to pdf and print in a single command; otherwise you will want to compile and print inside the loop or write a separate script. Make sure that somewhere on the page (possibly a footnote) is the index number, just in case. Be sure to save the random number generator seed that you initialize at.
Ms. Vegetable tells me that one could accomplish this same idea with VBA inside of word rather than using latex, but not the specifics.
posted by a robot made out of meat at 11:12 AM on January 8, 2012
Oh, I'd like to amend my previous post to include a standard stat consult cop-out:
The most important thing in worrying about question ordering is subject area knowledge. If you are really only interested in a single question and worried about participants tiring, then just demanding that question appear in balanced order might be ok. If you are concerned that particular questions will prime some others, balancing the respective positions of those is most important. If you are looking at associations between some questions, as long as the priming questions are consistently placed maybe it doesn't present much of a validity problem.
posted by a robot made out of meat at 11:29 AM on January 8, 2012
The most important thing in worrying about question ordering is subject area knowledge. If you are really only interested in a single question and worried about participants tiring, then just demanding that question appear in balanced order might be ok. If you are concerned that particular questions will prime some others, balancing the respective positions of those is most important. If you are looking at associations between some questions, as long as the priming questions are consistently placed maybe it doesn't present much of a validity problem.
posted by a robot made out of meat at 11:29 AM on January 8, 2012
Save the list as text in
posted by nicwolff at 12:43 PM on January 8, 2012
in.txt on any handy Unix or Mac OS box, then dofor f in {0,1}{0,1,2,3,4,5,6,7,8,9}{0,1,2,3,4,5,6,7,8,9}
do
perl -MList::Util -F'\n' -lan0e 'print for List::Util::shuffle @F' in.txt > $f.txt
done
posted by nicwolff at 12:43 PM on January 8, 2012
Heh, I outsmarted myself there: the third line could be just
posted by nicwolff at 12:48 PM on January 8, 2012
perl -MList::Util -e 'print List::Util::shuffle <>' in.txt > $f.txt
posted by nicwolff at 12:48 PM on January 8, 2012
« Older Fill my DVD queue with healthy-living flicks. | Camera Needed. Not too expensive, likes indoors. Newer »
This thread is closed to new comments.
This is pretty simple to do in Excel.
Since you say there are 8 questions and they're short sentences, just type the stuff into Excel. Once cell per question.
Then use the RANDBETWEEN() function (info) to generate a pseudo-random order of numbers between 1 and 8 inclusive for each of the 200 respondents.
Obviously, given 200 respondents and only 8 questions, some of your respondents' question orders will be the same.
posted by dfriedman at 9:11 AM on January 8, 2012