Share your IT infrastructure, hosting and version control thoughts for web development firms?

June 5, 2008 12:47 PM

I work for a small web development company. As we are growing, I have been tasked with looking at a few pieces of our business: - Version Control (Currently we don’t do it. We’re starting to use Subversion at my recommendation) - Development Environments (We don’t exactly have formalized dev/stage/production right now, more like just dev/production…) - Hosting (How we host our different clients varies… too much… we need a good solution) - Infrastructure (i.e. how we host our own email, how we back up our data) I already have some thoughts and recommendations, but they’re based on my personal experiences, so I thought I’d see what other experiences are out there and find out what other people/companies are doing…

A few notes:
- We’re a small shop – a little less than 25 people. We’ve been growing quickly and won’t have a dedicated IT person for probably another year. As a result, we’ve currently just got an Outlook server (as in.. not full blown Microsoft Exchange) and the inability to do things like share calendars, etc, is causing some pain. Any recommendations for Exchange hosting or alternatives would be welcome.
- We do both Windows (ASP) and Linux (PHP) based web stuff, so as far as dev/stage/prod environment suggestions go – we basically need to do double that. Most of our projects have involved only one or two developers at a time, so version control and loose environments haven’t really caused any huge problems [yet?]. I feel pretty strongly that we should take preventive measures anyway, but I’d love to hear your thoughts/experiences.
- Since we don’t have an IT person here, we’re hosting stuff for our clients on sort of a seat-of-our-pants basis. We’d been partnering with a hosting company to an extent, but they’ve proven extremely unreliable. I’d definitely be interested in hearing what other web shops do as far as hosting things for their clients. I like the idea of finding a solid provider to partner with, rather than trying to bring it all in house… but what do you do?
- Finally, I’m pretty impressed with our backup practices, for the most part. Machines are backed up daily to both a RAID and alternating sets of tapes that are taken offsite. However, network-based backup would probably be much better. Problem is, our network connection stinks (a single T1) – but we’re looking at moving to business class Cable, which these days is much faster (and cheaper). Any thoughts on this move would be appreciated.

posted by twiggy to computers & internet (10 comments total)

Version Control: Vault. Not quite Magic Versioned File System That We Don't Need To Worry About, but close. Vault wins for its tools, happy coexistence with NTFS and use of MSSQL and IIS, stuff we use all day. If we were a LAMP shop, we might think otherwise.

Email: Hosted Exchange through Appriver. Not 100% happy with their spam filtering, but much, much better than running our own.

Hosting: Rackspace dedicated servers. Expensive but insanely reliable. Pass the costs on to your customers. When we do consulting/custom dev work we recommend "real" hosting like RS over the cheapies. Customers who won't pay for mission critical web hosting aren't customers we want.

Backup: Horrifically complicated PKI encryption plus RSYNC process. One of those "don't mess with the script" things.

We encrypt with keys locally, then backup to an offsite server with rsync. Offsite database servers do the same, rysncing to the office and a third location. We do it over dsl but someday we will outgrow that.

Monitoring: a bunch of nagios scripts to make sure the backups all happen and the servers and services still work.
posted by mrbugsentry at 1:19 PM on June 5

Seconding Rackspace, you can defer almost all problems to their excellent tech support, so you won't have to bog down your own people for that. It is rather expensive though - some clients may balk at the price but like the previous poster said, these are not clients you want to have. These are the clients that call you up at 2 AM because they accidently deleted public_html with their ftp client.
posted by bertrandom at 1:42 PM on June 5

I third Rackspace, their support is awesome.

Backups: I've started adding redundant Mozy accounts to all the machines in our small office. Local backups are nice, but Mozy is the shit.
posted by crickets at 1:57 PM on June 5

FWIW, you can get cheap 1u rackmount server on ebay for $200 that will be fine as junk boxes for testing php sites. I do that for setting up local environments; I'd never use a junker for production, but it's just fine for making sure the code works- only has to support a couple of testers, max.
posted by jenkinsEar at 2:02 PM on June 5

I've used AppRiver Exchange Hosting with great success. They're easy to use, reliable, and very responsive. I happily recommend them.

Version Control: Subversion is great, you can lock people out of stuff for security, you can tie it into various build and test systems. If you don't care about any of that, you might like a distributed system like git or darcs instead, which have the nice feature that you can record a change at one point in time, but push it to the primary repository later. Thus allowing you to make patches even if you're not on the company network.

Hosting: Rackspace is probably fine if you don't have your own techies. That said, if you ever hire techies, you'll do better to colocate some HP DL360s someplace and buy the iLO licenses. (But that's something you can always do later.)

Environments: VMWare is your friend. It makes it a lot of potentially complicated tasks dead simple, since you can make sure that your dev environments really are *exactly* the same as your prod environments... can easily do roll forward/roll back without disaster, etc.

Backup: Do your damnedest to make it so desktops/laptops are burnable.
posted by Project F at 2:05 PM on June 5

Whatever you do as far as where/who's hardware it is you want the capability to reboot or turn on a machine remotely. In addition to installing the OS. Your IT guy will want this when he comes on. iLO as Project F suggested is the most popular option(as long as your provider is using HP machines). Or bring it in house and shell out some dough for a short term consultantation to get you up & running. You should at least be redundant on power, pipe & storage. Don't forget monitoring! A decent hosting provider should have an established system. Ask them what their monitoring archtecture is while you're shopping around. You want to know how they get notified, that they're looking at what YOU think is important and what turnaround time is.

Version Control: git + github to get a distributed location even seperate from your colocated / hosted site. SVN is nice too, more mature/widespread than git is.

Backups: Your hosting provider has a plan too, right? Ask them about it. Question them what it would take to come BACK from a backup. Say everything takes a nosedive, can you bring the business back from offsite storage? How long? Using git to github as a backup solution instead of rsync is also an option. Do it over SSH to protect your data over the wire.

Flesh out a dev, build, test release cycle. Don't leave out test! Get some friends or other non-invested(devs know their apps best and tend to be gaurded about them) people to hammer away on your apps at the least. If you're going the VM route try to at least keep your prod separate from your other environments as far as physically. That way when someone knocks out both power cables(redundant, right?) every env doesn't come crashing down.

Look for a cheap distributed CDN. Panther Express & Edgecast to name a few. Google and Amazon also have some options for content distribution.
posted by mnology at 2:29 PM on June 5

You might consider Google Apps over Exchange hosting. Exchange is great - don't get me wrong. But at least evaluate whether Google Apps may be simpler for your environment and users.

Seconding using VMware for your dev environments. VMware Server is free. It doesn't have the hot failover or live image backup that ESX does, but you can't beat it for the price. As long as you backup your images, you can use pretty much any kind of PC to run them.
posted by cnc at 4:18 PM on June 5

Backups, DataDomain - Works fantastic, scales well, can synch to remote DD's. It's a great product.

I'm a big advocate of OpenNMS, Cactii and managing your own environment.

Go VMware when you can, lease a half rack and get someone to own uptime for the environments, at 25 people you're large enough to have someone focus on infrastructure and ops as their job.
posted by iamabot at 4:42 PM on June 5

Version control/Staging: Each developer has a private Git repository on the dev server, which is hosted as a name-based virtual Web site so they can test their working changes. They can freely branch and merge features and fixes, and once one is stable it can be merged into the staging site's repository; once it passes regression tests there it can be pulled into production. Neither the staging nor production sites is ever edited directly.

Hosting: We've been hosting Web sites for 15 years and we've learned this lesson: don't own the hardware. We have managed servers at NTT America (they bought Verio). They do a terrific job both on reliability - four nines - and security - firewalls and ID/PS are managed by IBM's ISS unit. (We host public sites for tier-1 financial-services companies, so our hosting has passed white-hat penetration attempts.)

Features that have to scale dynamically are hosted in Amazon's EC2 cloud. They offer static IPs now, and you could actually host sites there, but we use them mostly for content processing.

Backup: Verio does tape backup for disaster recovery, but we do a nightly rsnapshot to a server at Easyspeedy in Denmark for immediate casual recovery of work files.

Infrastructure: IMAP to Google Apps for office e-mail, Basecamp for contact management. Our private wiki is on a Mac Mini at the office, which needs to change.
posted by nicwolff at 4:53 PM on June 5

Rackspace was great when I had a dot com, though now I'd look into Amazon's EC2, very expandable and bringing another system online config'd exactly like your other system is very easy.

As for mail, rather than Google Apps, since this was before then, we did our own with an IMAP server (accounts stored in a mySQL database with a web frontend).
posted by Brian Puccio at 8:11 AM on June 6