Are you a researcher or academic? I want to make neat software for you!

August 18, 2015 6:11 PM

Dear academics who use online questionnaires in your research: help me understand how you get those questionnaires online and go about using them. I want to help make them better.

I've participated in a number of online research surveys (mostly in the social sciences). Almost without exception, the sites are awful. They don't work well on mobile devices; basic functionality is half-broken; little or no attention has been given to UX or usability; they're just generally clunky and ugly.

I'm trying to understand why this is. I'm a web developer, and I'm looking for a side project to tackle. It occurred to me that a friendly, easy-to-use tool for creating these kinds of questionnaires could be just the thing.

Feel free to answer as generally as you like, but here are some more specific questions:

1. Do you typically use a third-party service (such as psychdata.com, which powers the example cited above), or do you get a [ student / intern / tech-savvy co-researcher / person from your university's IT department ] to set something up on your university's own servers? (I've seen some questionnaires on .edu domains, so I assume they're hosting their own software of some kind.)

2. Why do researchers (apparently) use their own purpose-built questionnaire software, instead of friendlier mass-market services such as SurveyMonkey? Are you afraid (probably justifiably) that a SurveyMonkey link won't be taken seriously? Do mass-market services lack features that you need? Are researchers just not typically concerned about usability?

3. Are there particular requirements (security, confidentiality, requirements imposed by your university or standards of research, etc.) which limit your choice of questionnaire software?

4. Assuming that a free (or cheap), high-quality, easy-to-use third-party service was available: do you think researchers in your field would be willing to adopt it? Or do you think a lot of them are set in their ways, and unlikely to get adventurous in this area?

5. What neato features would make your ears perk up w/r/t such a tool? The ability to export data in particular formats? Built-in data analysis tools? Something else?

Again, these questions are just to get the juices flowing—I'm really just asking you to walk me through a typical process of setting up an online questionnaire as part of a research project, and what considerations guide your decisions during that process.

Thanks!

posted by escape from the potato planet to Science & Nature (18 answers total)

I apologize in advance if this sin't answering the question, but REDCap is a free software solution that an organization I work with uses. There are a bunch of training videos here that illustrate some of the functionality that's useful when putting together a survey project.
posted by MadamM at 6:27 PM on August 18, 2015 [4 favorites]

we're doing this now, one giant headache is making it easy for people to save their work mid-survey and return easily.
posted by mercredi at 6:34 PM on August 18, 2015

I'm more quantitative than most psychologists. I'm mostly going to answer #2. My current lab rolls our own in Django. I've used Qualtrics in the past, and even that is kludgy and overly-simplified. Mturk is heavily used in the social sciences nowadays, so easy integration there (approving HITs, screening workers) is a huge plus.

I need to be able to upload a CSV or JSON that I've generated programmatically on my machine in Matlab or Python rather than a GUI for constructing items (and a template for creating it based on the options in your system would be essential). After that, I need real counterbalanced randomization and pseudo-randomization based on individual items, blocks of items (which may or may not be randomized within blocks), or item "flows" with solid logic patterns.

As an example, say I have ten types of items, A through J. I want to give all of them 8 times each, but split within 4 blocks, so each block contains two of each. I don't want any of the same item types to be within two items of each other. Each of the four blocks needs to start with a different type. Across all my subjects, I want to test every possible order of start item. Oh, and they're all videos that need to be shown at a specific frame rate.

That's worst case scenario, but it's the sort of thing that makes people code their own. If you can do something like that, and be less time consuming than coding by hand, then yeah, I think you'd get people using it :)
posted by supercres at 6:35 PM on August 18, 2015 [3 favorites]

At the university where I review research, a small number of survey software options have been thoroughly vetted by information security staff to be acceptable in terms of both the security required by the university, and the human subjects security protections required by the institutional review board. The vast majority of our researchers use these tools because unless you have some major research need that absolutely can't be filled by the approved tools, it is not worth the series of major hoops you'd have to jump through to use something else. Also, the university has a site license for these software systems, vs. the expense of purchasing a license for your own use, or paying your staff to develop something unique.
posted by Stacey at 6:38 PM on August 18, 2015 [4 favorites]

Oh, and #3, within reason, isn't an issue since they sign an IRB notice.

Built-in data analysis tools?

Oh god no.
posted by supercres at 6:46 PM on August 18, 2015 [4 favorites]

#2: SurveyMonkey is not truly anonymous
posted by yeolcoatl at 6:47 PM on August 18, 2015 [1 favorite]

2. Because surveymonkey sucks and so do lots of others. SurveyMonkey can't do skips (if they said x to this quuestion show quetion 56 otherwise don't. If they said Y to question 27, skip questions 35-40) etc. They SAY they do, but in fact all you can do is show a page or not based on the immediately preceding question. SurveyMoney can't take a list of things entered by the R and plug them into other questions. Also, the method of creating questions in SurveyMonkey is clunky.

3. Our ethics board wants encrypted transmission and storage of data and gets antsy about data being stored in or passing through the U.S. due to the patriot act, which makes it impossible to ensure confidentiality if the data are in the U.S.

5. You can't do built-in data analysis. Not serious built in data analysis. I'm sure you could set something up to run a frequency or crosstab or correlation, but that's not publishable analysis. Just give us data we can analyze in real stats software. You will want to have some method for users to create userIds and passwords. You want to NOT put these in the data. Nor do you want to put into the data things like IP address (surveyMonkey does this, I believe). Finally, killer app: Handle honoraria! If you can mail or paypal respondents money, that would make everyone's life easier. However, the person's contact information must be kept completely separately from the data.
posted by If only I had a penguin... at 6:55 PM on August 18, 2015 [2 favorites]

If you can make it export well to Python, MATLAB, and R, you have satisfied many people from the data perspective.
posted by oceanjesse at 7:11 PM on August 18, 2015 [1 favorite]

Oh, something else that is small and probably very easy to do, but could make life simpler: Have an option to have yes/no variables and other variables with two values coded 0/1 instead of 1/2. A 0/1 coding is what you need for a logit model or to use those variables as dummy variables. Yeah, they can be recoded, but wouldn't it be nice if they just came set up right in the first place?
posted by If only I had a penguin... at 7:23 PM on August 18, 2015 [1 favorite]

I'm a faculty member/researcher. My university has its own system that it would like us to use and it is free and is secure/meets all the IRB requirements.
Yet it sucks, especially on mobile.
MANY universities contract with Qualtrics for site licenses and in my most recent survey, I had a grad student evaluate a bunch of the different platforms and we decided Qualtrics was the best deal - fairly cheap, met all the IRB security requirements, looked good on mobile, exported to all the different formats the team needed (R, STATA, SPSS, Excel) and in my case worked with non-Latin/English characters, and everything is/was hosted in the US, which matters because I'm collecting data from very vulnerable people who are under a lot of surveillance. Qualtrics also lets you have a fairly high N for free.
Most of my researcher friends prefer Qualtrics right now. SurveyMonkey is popular with undergrads, but I don't see it used by my scholarly friends very often anymore. There was another one that had a name that involved a fruit that a few people used a few years ago, but not that often.
People that need to buy samples from the survey provider also do this via Qualtrics or SurveyMonkey. Last I checked (a few years ago), SurveyMonkey sold much cheaper American representative samples than Qualtrics.

For you as a web developer, I'd suggest the following barriers:
- IRB requirements are serious business and a lot of university IRBs have a list of preferred/understood/known providers
- this stuff is SO reputation-based... if some well-regarded researchers trash your tool on Facebook, you're dead to their networks forever
- you'd want to work with a survey research firm to know what sort of design elements people want (SKIP LOGIC FTW)
- there are so many different output formats people want, and that updates regularly, and no one really cares about live automatic data analysis from the provider
- you may eventually need to set up a payment system where it is easy for us researcher to pay you via a PO
- if you ain't selling data, you're missing out on a key part of this market
posted by k8t at 7:38 PM on August 18, 2015 [1 favorite]

Mod note: Removed a link from the post with the asker's permission to avoid potentially mussing up someone's research, carry on.
posted by cortex (staff) at 8:17 PM on August 18, 2015

Authentication and security need to meet one standard for most basic research, but for anything with a health data element you often need to meet HIPPA/HITECH standards too. That means certain encryption standards and using unique non-identifying tracking codes and passwords to authenticate each participant, so that survey results can be linked to other data (health imaging, genetic results, future surveys). You might need secure email invitations with individual links. The ability to easily set up and administer longitudinal surveys set at certain intervals, or based on the results of the last survey, would be helpful too. RedCap and Qualtrics do some/all of this so that's one reason why people with complex protocols stick with them.
posted by slow graffiti at 9:33 PM on August 18, 2015 [2 favorites]

I use Qualtrics through a licence my department pays for. I like that the interface is easy to use but also has always been able to accomplish whatever things I want to include (skip logic, complicated randomization set-ups, and supports Javascript if I need to include something in a question that their built-in options can't do). I also really appreciate their service because customer service is fantastic - whenever I have run into an issue, they have responded in less than 24 hours and have actually solved whatever the issue was. I would probably be skeptical that a free service would be able to be that responsive.

I would honestly also be pretty skeptical about how a free/super cheap service was making their money...are the putting ads in my survey? I wouldn't want that since I can't control what is shown and I need that for my results to be valid. Are they somehow collecting/using/selling user data? I feel like that REALLY would not fly with IRB requirements. I'm not really sure how else "free" companies make their money, so I'd be suspicious unless this is very clearly laid out in the promotional materials and makes sense as a business model.

And finally, a big benefit of Qualtrics is that all of my old surveys are achived pretty much indefinitely so if I want to repeat a survey and tweak some questions, it's super easy to copy it over, and if someone has a question about exact wording/implementation/etc. I can go back and double check. It is very easy to transfer accounts between institutions when you change jobs, and they're an established company so I trust they'll be around for the long haul. Not sure how well I would trust some brand new company to store my data indefinitely so I can go back and retrieve old surveys later.
posted by rainbowbrite at 6:39 AM on August 19, 2015 [2 favorites]

Came to say what slow graffiti said. It's all the back-end functionality and the IRB/HIPAA considerations that make it easy to use institution-approved software. Another vote for REDCap! (It includes options to set up the survey so that users can quit halfway through and return to finish. It also let's me resend the customized link to nudge those who only partially finished to return and complete it -- whether I want to do that from an ethics/science perspective is another question.)

To be fair to modern day researchers, paper and pen researchers didn't give a lot of thought to how ugly their paper surveys looked, either!
posted by vitabellosi at 8:45 AM on August 19, 2015

I'm a rehabilitated academic, now working in policy for a non-profit that does a lot of work ferrying data and analysis between industry/researchers and regulators. I am baffled that survey monkey is the go-to way to do this, even for U.S. federal agency centers. So:

1. Yes, usually survey monkey (if we're working with a partner who already insists on using it) or a form built in WordPress (apologies for not knowing what these are called, our in-house web team makes them for us--does Lightning Forms sound right?).

2. Yes, I cringe when I tell someone to take our survey, which is on survey monkey. It's friendly and usable, but almost cartoonishly so. Does that make sense? What if I want to make something look a particular way, just black text on a white field? Or display three grouped questions at a time? Or display some complex figure alongside a particular question?

3. Dealing with open-ended responses is complicated without turning a survey into a validated research tool. It'd be great to be able to do some front-end work that would help automatically code or group open-ended responses in more than one way at a time. It'd be great, for example, to see the output as (1) a word frequency count, (2) correlation of responses using words x, y, z by respondent's country of origin, (3) number of time that a respondent who used word X in question 7 also answered no back in question 2. That sort of thing.

4. Yes! We would try anything new if it could be vetted by our internal group for security. Also, if this is something that could be integrated with a WordPress site, we might use it tomorrow.

5. See #3.
posted by late afternoon dreaming hotel at 8:54 AM on August 19, 2015

We use surveymonkey, even though it's not fantastic, because our college has < 700 students and large data collection companies don't scale down well. Qualtrics offered a $2500/10 users license, whereas surveymonkey runs around $300/year. We don't have the tech support available to try something like REDCap, so that's out. Before surveymonkey could handle random assignment to conditions, I used a small in-house program through an institution in the UK that I'd helped beta test. So, cost, and scaling down costs are most important to me.
posted by bizzyb at 5:11 PM on August 19, 2015

I'll nth the IRB issues.

Unlike some of the other posters, I dislike Qualtrics. Maybe not immensely, but it's definitely dislike. I find I wrestle with it anytime I need it to do something I don't already know how to do. No experience with anything else other than Surveymonkey. You already know all about that one in terms of design etc and participant response.

One thing to potentially think about, in a helpful way, is whether your design could:
--(a) increase the overall response rate;
--(b) increase the response rate from certain groups (who don't use e.g. Surveymonkey but would use yours);
--(c) otherwise change the dynamics of who participates (alter the composition of survey populations);
--(d) make it easier to use as guided/structured interview tool for surveyors; or
--(e) make it a superior tool to give to survey participants, for example on a tablet, which is basically just a way for in-person surveying to not have to do data entry;
--(f) be better considered than others when surveying non-WEIRD populations, because Qualtrics, Surveymonkey, etc are definitely for the WEIRD.

Hope you can make better tools for us!

(There might actually be other uses I'm not thinking of, depending on field.)
posted by migrantology at 8:58 PM on August 19, 2015

An additional reason why my group sometimes makes our own apps is because in some cases, we're deploying interviewers with tablets to places where they may not have an internet connection. So we make an app that will allow for the storage and collection of data which will be uploaded when they can next access a secure connection.
posted by metasarah at 12:54 PM on August 21, 2015