Online petition system that can handle 20,000 submissions in two days?

August 1, 2014 4:55 PM

We have a Drupal website with the standard webform module that we generally use to handle online petitions that our nonprofit routinely does. It can handle maybe a few hundred petition signers in a day without putting us too badly over our shared hosting CPU/SQL limits. Now we need to handle 20,000 signers in a couple of days.

We need to collect maybe 6 or 7 fields (a comment, name, address, city, state, zip, etc) and a checkbox or two. It would be approx. 20,000 individual submissions, maybe 30,000, and I anticipate most of them coming over the course of two days.

Results could be saved in a file, an SQL database, or any other convenient form that we can download somehow.

Any thoughts about how to do this in the simplest and most cost-effective way?

Maybe just a simple lightweight php script that saves form info to a flat file or a sql db, hosted on our shared server? Could that kind of setup handle 500-1000 responses per hour for a couple of days straight, along with maybe a few higher bursts?

Or some kind of externally hosted solution like wufoo forms?

This is a one-off situation, and the solutions I've found so far that are customizable and capable of handling 20,000+ responses are all priced as though you'll be gathering 20,000 responses/week forever, rather than once for two days. Or, they are too generic and don't allow us to specify the fields we want to collect, or they don't show us to download the results.

We be willing to pay, but ideally more like $25-$100 rather than $250-$1000.

Any solutions out there?

posted by flug to Computers & Internet (12 answers total)

I'm pretty sure you can do this in Google Forms. I'm checking and I don't see any sort of submission limit, though there is a spreadsheet size limit that I can't imagine 20-30K responses would hit, but you could clear it out every 12 hours if you're nervous. (Form responses get dumped into a spreadsheet, 1 response per row -- it's very friendly to then do whatever you want with it). They're not infinitely editable but the sort of thing you're talking about it should be completely doable. You can make a Google Form and embed it on your site.
posted by brainmouse at 4:59 PM on August 1, 2014 [2 favorites]

there is a spreadsheet size limit

The limit is number of cells in the underlying spreadsheet (i.e. number of replies × number of fields in the form), and the current version supposedly handles up to 2 million cells.
posted by effbot at 5:08 PM on August 1, 2014 [1 favorite]

Have you contacted your shared hosting provider to see how much bumping your current bandwidth limits up for a month would cost?
posted by rockindata at 5:21 PM on August 1, 2014 [1 favorite]

A few hundred signers per day puts you near your limit? That is fairly mind-boggling, and suggests to me that you probably have a problem with your SQL somewhere. Do you have someone who can check your code and database?

I have done some web form submissions via google forms, and I gotta say that I wouldn't really do anything mission critical that way. It's too easy for someone to accidentally hork all your data by messing around directly in your spreadsheet. That goes triple if you've got more than one person in your petition pie. ("Wait, what? You told the intern to rename the column "Zip" to "Zip Code"?!? Now the form isn't mapping to the spreadsheet correctly any more, and we are halfway through the collection period!")

I'm not super informed about petition software , but would one of the several survey software solutions (e.g. SurveyMonkey) work for you? They are relatively inexpensive subscriptions, and probably have well-defined performance metrics, so it should be easy to tell if they would meet your needs. I expect there are probably similar tools for petitions, specifically, though I am more familiar with SalsaLabs (higher end solution) and change.org, which...well, did you check out change.org? Do you need to host locally, or would that work for you? I don't know a lot about them, except that I seem to get a billion "change the world" petitions through them, and they are certainly big enough to handle 20k responses.
posted by instamatic at 7:05 PM on August 1, 2014

I would think it would be cheaper (once you factor in time to set up an alternate system) to upgrade your hosting temporarily. I also am surprised that a few hundred forms per day is putting you anywhere near any kind of limit. Either your shared hosting is very limited or something is very resource intensive that shouldn't be.
posted by ssg at 11:04 PM on August 1, 2014 [1 favorite]

If I understand your situation correctly, I think the solution is to move to Blackbaud or Salsa or NationBuilder, which are capable of not only handling the server load, but give you CRM tools to manage the data you collect.

I'm confused as to where you expect to get those kinds of numbers, if not from emailing a list you already possess. In any case, with a list that size, cost is no object, because you have a sizable pool of potential donors. Your digital strategy should not cost you anything at all. The $300/mo NationBuilder charges for 30k emailable individuals will be dwarfed by what you take in in donations when you hit that list up for support.
posted by univac at 2:59 AM on August 2, 2014

ssg, I'm thinking that if something is broken enough to take them over a regular "cheap" hosted account's limits (is that something like between $2 and $20/month, say Bluehost or Dreamhost?), it's unlikely that upping their plan for a month would get them to 20,000.

But definitely you could talk to a sales or support person at your hosting company. Maybe your account is somehow different than the ones I've, uh, busted through with high load and/or borked SQL. (Source: many hours long ago on the phone with sales and support while running a webapp that was both too damn big for shared hosting *and* with a couple of pieces of hidden bad SQL in it. When it went over limits, the hosting company's monitoring services took it offline automatically until load normalized. Repeatedly. Though only after it dragged everyone else on the shared server down with it. It's, uh, been a long time since I've made that mistake, in my defense.)
posted by instamatic at 6:38 AM on August 2, 2014

When you're talking about Drupal, god only knows what terribleness you've got running. A few hundred per day is an unsurprising place to start seeing scaling issues with tools like Drupal or WordPress.

Use google forms or survey monkey or wufoo -- any of those (and probably a dozen others) will have no problem handling the load you're talking about.
posted by toomuchpete at 9:59 AM on August 2, 2014

Limesurvey immediately comes to mind. It's open source and free, so you can host it yourself. The site offers a demo on their server, if you want to kick the tires.
posted by Pronoiac at 5:20 PM on August 2, 2014

Response by poster: Thanks for all the helpful advice and thoughts.

Just to clarify a couple of things: It is a very large site in terms of page numbers (somewhere over 5000 pages--blog and news entries going back years, etc.--), somewhere over 5000 pages and the site gets a decent amount of traffic, say 5000-10,000 pageviews a day.

It's cached and optimized to the hilt using Drupal's built-in caching, the Boost module (caches everything in static files for anonymous users), and Cloudflare, which is the only way you can carry that amount of traffic on a shared hosting type environment. Despite all the caching, it probably catches only 2 of 3 page loads or so--the rest are things that can't be cached like form submissions, or files that aren't in the cache yet or that have expired.

Also, the Drupal site is backed up by CiviCRM.

So our current setup and current traffic level puts the CPU & SQL load at just under our account limits for most days. This is on a hosting account that is shared but is also setup/marketed specifically for folks running drupal/civicrm sites.

So I'm guessing that if we suddenly quadruple that traffic (say 10,000 signers/day, each petition signing would load two static pages and one dynamic pages, so that is similar to our existing traffic but 3X as much) we'll be w-a-y over our account limits for a couple of days.

I'm also guessing that if we handle the the petition signing by way of a simple static HTML page that submits the form to a simple PHP page that does one simple SQL write to record the data and spit out a message that says "Thanks for signing!", that we'd be able to handle that amount of traffic with ease.
posted by flug at 11:34 AM on August 3, 2014

Response by poster: Also, if we were going to be able to use this size list regularly for membership/donor development, believe me we'd be up for the $300/month big metal server & software, but a good 95% of the signers will be outside the geographic area of our group.

It will be very helpful to have those folks on board this particular petition, but I doubt the conversion rate for these folks who are outside our geographical area would be high at all. So once this one-time thing is finished, we're likely to be back to our normal situation, which annoyingly is just at the limit of what the shared hosting type service can handle but just below the point where it really makes financial sense to take the dive for the $multi-thousand/year hosting & CRM plus a $many-thousand conversion over to the new CRM & system.
posted by flug at 11:36 AM on August 3, 2014

which is the only way you can carry that amount of traffic on a shared hosting type environment.

This is an assumption you probably need to re-evaluate. You're talking about, on average, a hit every 10 seconds. That's well within the capabilities of a shared hosting setup unless a) it's very, very bad or b) someone has done something very, very stupid with Drupal.

If you're sure (b) isn't the problem, I'd look into a new host.
posted by toomuchpete at 9:44 PM on August 3, 2014