How much would a basic online database for environmental data cost?
January 8, 2010 6:27 AM Subscribe
What would the rough cost (as time x rate) for developing a database for basic environmental data (temperature, water level, fish counts) and metadata (site, date, time) with an online form for volunteer data entry?
I am working on a grant that entails collecting basic environmental parameters (temperature, water level) from automatic dataloggers and fish counts from volunteer observations. The datalogger data is going to be automatically downloaded into a spreadsheet from the logger, while the volunteer data will be collected on data sheets in the field and then will be entered into an online form.
I am currently trying to figure out the budget, and I am totally lost on how much the database/website combo is going to cost. The few rough estimates from friends who do database work have come in around $6,000 (~120 hrs @ $50/hr) but I have no idea if that's reasonable. I am also faced with a granting agency that would rather see more money go into the database side of things than the website, and the two "halves" need to be split up on the budget.
Database requirements:
- has to hold records on the order of 10,000-100,000, but only of a few types of data
- has to be able to have data imported (loggers) as well as contain input from the online form
- has to be able to be exported for analysis
Web form requirements:
- relatively user-friendly
- does not have to be beautiful, just functional
- ability to choose sites via a map would be a plus but a list of sites (dropdown?) would be just as good
- must allow entry of multiple data points at a time, preferably 10-20 at a time.
- ideally (and I realize this bumps up the cost) the site would display the data (i.e. choose a site and see a graph of recent temperatures)
Both the website and database are going to be hosted on existing web space, so that is not an issue.
I am working on a grant that entails collecting basic environmental parameters (temperature, water level) from automatic dataloggers and fish counts from volunteer observations. The datalogger data is going to be automatically downloaded into a spreadsheet from the logger, while the volunteer data will be collected on data sheets in the field and then will be entered into an online form.
I am currently trying to figure out the budget, and I am totally lost on how much the database/website combo is going to cost. The few rough estimates from friends who do database work have come in around $6,000 (~120 hrs @ $50/hr) but I have no idea if that's reasonable. I am also faced with a granting agency that would rather see more money go into the database side of things than the website, and the two "halves" need to be split up on the budget.
Database requirements:
- has to hold records on the order of 10,000-100,000, but only of a few types of data
- has to be able to have data imported (loggers) as well as contain input from the online form
- has to be able to be exported for analysis
Web form requirements:
- relatively user-friendly
- does not have to be beautiful, just functional
- ability to choose sites via a map would be a plus but a list of sites (dropdown?) would be just as good
- must allow entry of multiple data points at a time, preferably 10-20 at a time.
- ideally (and I realize this bumps up the cost) the site would display the data (i.e. choose a site and see a graph of recent temperatures)
Both the website and database are going to be hosted on existing web space, so that is not an issue.
Best answer: I would strongly suggest trying to use off-the-shelf stuff for this.
You have two distinct problems, as far as I can tell:
1) Storing all this data from loggers, volunteers, etc. and having to analyze it. 100k records is not that much. Can you just keep this in Excel? I would seriously try to use Excel for this before paying someone to build and maintain a database. It will be so much easier.
2) Collecting data from volunteers on the web. Sites like Wufoo make it extremely easy to build a simple form that users can enter data into. They maintain the database backend, etc. You log into your Wufoo account and you can export all the data in a variety of formats. If you use Excel for your other problem, it would be pretty easy to export from Wufoo and just paste the volunteers' data into the main Excel workbook.
There are other sites that might work for you if an approach like this doesn't work: DabbleDB, etc.
posted by jeb at 7:11 AM on January 8, 2010
You have two distinct problems, as far as I can tell:
1) Storing all this data from loggers, volunteers, etc. and having to analyze it. 100k records is not that much. Can you just keep this in Excel? I would seriously try to use Excel for this before paying someone to build and maintain a database. It will be so much easier.
2) Collecting data from volunteers on the web. Sites like Wufoo make it extremely easy to build a simple form that users can enter data into. They maintain the database backend, etc. You log into your Wufoo account and you can export all the data in a variety of formats. If you use Excel for your other problem, it would be pretty easy to export from Wufoo and just paste the volunteers' data into the main Excel workbook.
There are other sites that might work for you if an approach like this doesn't work: DabbleDB, etc.
posted by jeb at 7:11 AM on January 8, 2010
As far as I know, $50 / hour is actually _low_ for web development that involves programming. It's possible that le morte de bea is thinking of British pounds, or maybe just web design (although that still seems low to me).
Best of luck.
posted by amtho at 7:34 AM on January 8, 2010
Best of luck.
posted by amtho at 7:34 AM on January 8, 2010
I'm a database and web developer in Vancouver, BC, so this is right up my alley. I charge $75/hour, but I tend to quote in lump sums so that I can have a sliding scale depending on the customer. I would ballpark your project at around $4-5,000 for me to execute, split 50/50 between the database and the site, respectively. However, it's been my experience that people asking for such quotes rarely provide sufficient detail up front, leaving out things like "oh yeah, we need an automatic way to create accounts for the volunteers", so $6,000 doesn't sound unreasonable at all.
Depending on what you're willing to accept, you could do something like this for as little as $2-3k for a simple bare-bones system.
posted by fatbird at 8:00 AM on January 8, 2010
Depending on what you're willing to accept, you could do something like this for as little as $2-3k for a simple bare-bones system.
posted by fatbird at 8:00 AM on January 8, 2010
Best answer: A website that simply presents a form, and adds he results as entries in a database table would take *way* less than 120 hours to develop. And exporting data from MySQL to CSV is easy.
Creating features to let you manipulate & graphically view data in the web application would take longer to develop.
If your estimated development cost is $6,000 an off-the-shelf solution will almost certainly be cheaper. Since you intend users to upload data from spreadsheets, DabbleDB (which supports graphing, CSV export etc.) might be a better choice than Wufoo. An account for three users costs $24 dollars a month.
posted by James Scott-Brown at 8:02 AM on January 8, 2010
Creating features to let you manipulate & graphically view data in the web application would take longer to develop.
If your estimated development cost is $6,000 an off-the-shelf solution will almost certainly be cheaper. Since you intend users to upload data from spreadsheets, DabbleDB (which supports graphing, CSV export etc.) might be a better choice than Wufoo. An account for three users costs $24 dollars a month.
posted by James Scott-Brown at 8:02 AM on January 8, 2010
Response by poster: Thank you all for your insight. I've talked to a few programmer friends and they suggested erring on the side of caution (i.e. build in a buffer for contingencies), but given what those of you who gave me estimates said, it sounds like $6,000 (split up) would probably be sufficient.
That said, those of you who suggested sticking with Excel and using an off-the-shelf solution have piqued my interest. I am not sure that solution would fly with my colleagues but I *personally* prefer to work in Excel and given that I will undoubtedly be left in charge of managing the data, the idea sounds great to me.
I probably should have provided an example of the sort of thing I might be looking to do; does this site still look like it would be $6,000? This is a site that does something a little similar to what we're looking to do. Unfortunately you can't log into the volunteer access page, but it essentially just has labeled boxes and a submit button.
posted by nekton at 8:48 AM on January 8, 2010
That said, those of you who suggested sticking with Excel and using an off-the-shelf solution have piqued my interest. I am not sure that solution would fly with my colleagues but I *personally* prefer to work in Excel and given that I will undoubtedly be left in charge of managing the data, the idea sounds great to me.
I probably should have provided an example of the sort of thing I might be looking to do; does this site still look like it would be $6,000? This is a site that does something a little similar to what we're looking to do. Unfortunately you can't log into the volunteer access page, but it essentially just has labeled boxes and a submit button.
posted by nekton at 8:48 AM on January 8, 2010
Just using Excel would be a bad idea when what you're really talking about is controlling data entry for a variety of geographically distributed people. If you want to work with the data in Excel, it's nearly trivial to export a database table to a spreadsheet. Online systems like Wufoo and such could work, but they limit future development to what Wufoo will support, which might not be the direction you want to go.
Your best bet right now is to sit down and actually write a spec saying exactly what you want--that's how you keep a developer's costs in line. When I hear a verbal proposal I always assume there are things being left out simply because they're not apparent at the time; it's also been my experience that getting exactly what you want from a developer inevitably leads to "hey, can we do this as well?", which is why I'm cautious in my estimates in my favour. A contingency buffer is a very good idea.
posted by fatbird at 9:02 AM on January 8, 2010
Your best bet right now is to sit down and actually write a spec saying exactly what you want--that's how you keep a developer's costs in line. When I hear a verbal proposal I always assume there are things being left out simply because they're not apparent at the time; it's also been my experience that getting exactly what you want from a developer inevitably leads to "hey, can we do this as well?", which is why I'm cautious in my estimates in my favour. A contingency buffer is a very good idea.
posted by fatbird at 9:02 AM on January 8, 2010
I would guess that web development (not design - I'm talking full-blown database-driven web applications) must cost a lot more in the US. I know plenty of freelance web developers here in the UK, all experienced people, who would leap at the chance of a £30 an hour gig (yes, I did the conversion). An average salaried web developer here will earn in the region of £15 - £20 an hour ($25 - $32). Freelance developers tend to charge a bit more to cover any down time, but I'd be out of business pretty quick charging $50 an hour.
Could be an exchange rate thing though - maybe you could look at sourcing the work overseas...
posted by le morte de bea arthur at 11:43 AM on January 8, 2010
Could be an exchange rate thing though - maybe you could look at sourcing the work overseas...
posted by le morte de bea arthur at 11:43 AM on January 8, 2010
wow. good web developers in america would easily charge four, five, six times that rate, leMdBA. Maybe you should start trying to pick up US-based clients.
The reason I'm suggesting things like Wufoo isn't that building an app thats in any way like what you are talking about is particularly hard to complicated, I just think its incredibly unlikely that you will actually get the app you want if you aren't used to managing software dev and you are working with a contractor. People are basically terrible at knowing what they want. That's why real software development teams mock things up, do prototypes, do testing, etc. The hard part isn't building it, its knowing what to build. You may think you've got a good handle on it now, but it is incredibly likely that once you've got your stuff set up you'll realize its wrong, the developer will not have understood what you wanted, you will will think perfectly obvious things didn't need to be explained and the developer will do things differently, your needs will change, something will break, etc. That's the stuff that will sink you, not building a simple form-'n'-db webapp.
If you use Excel as the master data store, and you realize that you need to start collecting another field, fine, just start doing it. Handle the old data however. With your web app, you've gotta call the developer, have him add the field to the DB schema, update the web app, migrate the old DB to fit the new schema. It's a big PITA. You'll need a new backup strategy to dump the database periodically, and you'll need to know how to restore from it. With Excel, you just do "Save As" and back up the files like you back up the rest of your files. If your webapp mysteriously goes down, who are you going to call? The developer probably isn't thinking he's responsible for adminning your app. Either way, he's not on 24/7 call, so your volunteers have to sit around twiddling their thumbs and hoping they don't miss any data until the site comes back up. A company like Wufoo has a whole team on top of stuff like this.
posted by jeb at 2:12 PM on January 8, 2010
The reason I'm suggesting things like Wufoo isn't that building an app thats in any way like what you are talking about is particularly hard to complicated, I just think its incredibly unlikely that you will actually get the app you want if you aren't used to managing software dev and you are working with a contractor. People are basically terrible at knowing what they want. That's why real software development teams mock things up, do prototypes, do testing, etc. The hard part isn't building it, its knowing what to build. You may think you've got a good handle on it now, but it is incredibly likely that once you've got your stuff set up you'll realize its wrong, the developer will not have understood what you wanted, you will will think perfectly obvious things didn't need to be explained and the developer will do things differently, your needs will change, something will break, etc. That's the stuff that will sink you, not building a simple form-'n'-db webapp.
If you use Excel as the master data store, and you realize that you need to start collecting another field, fine, just start doing it. Handle the old data however. With your web app, you've gotta call the developer, have him add the field to the DB schema, update the web app, migrate the old DB to fit the new schema. It's a big PITA. You'll need a new backup strategy to dump the database periodically, and you'll need to know how to restore from it. With Excel, you just do "Save As" and back up the files like you back up the rest of your files. If your webapp mysteriously goes down, who are you going to call? The developer probably isn't thinking he's responsible for adminning your app. Either way, he's not on 24/7 call, so your volunteers have to sit around twiddling their thumbs and hoping they don't miss any data until the site comes back up. A company like Wufoo has a whole team on top of stuff like this.
posted by jeb at 2:12 PM on January 8, 2010
Wow, really? I charge $150 an hour and I'm constantly in demand, even thinking of increasing my rates. I do e-commerce work, which (by design) inflates the cost, but $50 per hour? I'd starve or go get a full time job (shudder).
FWIW, I also do work for UK clients, and the $150 doesn't seem to faze them. Personally, I think it is 25% "expert credentials", and 75% asking for it with a straight face.
120 hours for a web form putting data into a DB is crazy, unless it was some horrible windows solution, then I wouldn't touch it. Even if I was also building in registration and setting up the server, I'd think it was more like 20. 30 at the outside.
As an example, last year I did a project which involved direct mail (postcards) with unique registration codes to a couple thousand recipients. They got amazon gift certs for registering. Lots of tracking of users registered, etc, all to a database. Total cost, including my brokering the Amazon gift certs was about $5K. Not tooting my horn here, just giving an example why the hours seem way out of whack. Thing is, I don't even think I'm the "fastest dev around", I'm careful and medium-speed, IMHO.
Advice, use a good framework, it will make everything so much faster and less painful. Django is my favorite, but if you only know PHP (please learn some other languages), then Codeigniter.
posted by Invoke at 2:20 PM on January 8, 2010
FWIW, I also do work for UK clients, and the $150 doesn't seem to faze them. Personally, I think it is 25% "expert credentials", and 75% asking for it with a straight face.
120 hours for a web form putting data into a DB is crazy, unless it was some horrible windows solution, then I wouldn't touch it. Even if I was also building in registration and setting up the server, I'd think it was more like 20. 30 at the outside.
As an example, last year I did a project which involved direct mail (postcards) with unique registration codes to a couple thousand recipients. They got amazon gift certs for registering. Lots of tracking of users registered, etc, all to a database. Total cost, including my brokering the Amazon gift certs was about $5K. Not tooting my horn here, just giving an example why the hours seem way out of whack. Thing is, I don't even think I'm the "fastest dev around", I'm careful and medium-speed, IMHO.
Advice, use a good framework, it will make everything so much faster and less painful. Django is my favorite, but if you only know PHP (please learn some other languages), then Codeigniter.
posted by Invoke at 2:20 PM on January 8, 2010
Well, £15 - £20 plus health insurance, an office, and paid vacation would be more attractive, I guess...
Still, remind me not to move to the UK without checking first.
posted by amtho at 2:46 PM on January 8, 2010
Still, remind me not to move to the UK without checking first.
posted by amtho at 2:46 PM on January 8, 2010
Hmmm - does Google have an online Excel equivalent - could you then use that to collect field data and then export to Excel?
I will tell you that trying to merge incoming Excel spreadsheets would be ugly, time-consuming and error-prone.
Actually - it looks like Google Docs does have a spreadsheet program, you can create forms and download the results in Excel.
At that point you may not need database or web-programming.
posted by jkaczor at 5:22 PM on January 8, 2010
I will tell you that trying to merge incoming Excel spreadsheets would be ugly, time-consuming and error-prone.
Actually - it looks like Google Docs does have a spreadsheet program, you can create forms and download the results in Excel.
At that point you may not need database or web-programming.
posted by jkaczor at 5:22 PM on January 8, 2010
jkaczor, Google Docs does let you create forms for data entry, but the maximum spreadsheet size would probably rule out its use. The actual limits are: "Each can be up to 256 columns, 200,000 cells, or 100 sheets". Specifically, the limit on the number of cells would be a problem given the aim of holding `10,000-100,000 records'
posted by James Scott-Brown at 9:12 AM on January 9, 2010
posted by James Scott-Brown at 9:12 AM on January 9, 2010
This thread is closed to new comments.
1. $50 an hour is about twice my hourly rate (and mine is probably towards the high end for UK web developers, at least outside London).
2. 120 hours isn't that unreasonable, but personally I would probably have quoted you around 80.
This isn't an offer to take on the work though :)
posted by le morte de bea arthur at 7:04 AM on January 8, 2010