Can the Mac Mini, the VW Beetle of server clusters?
August 25, 2006 3:49 PM Subscribe
I am looking toward upgrading my website's hardware. I want reliability and easy expansion.
Could migration to a cluster of mac minis be a good solution for a distributed website?
I would be configuring/building the servers in the UK and shipping to their server farm in the US. Basically I was thinking of using mac minis as they are standardised.
My theory is that the cost of adding more servers is relatively low and I will be able to devote new servers to specific sections that need more grunt.
e.g.: Tomcat servers, file servers, database mirrors etc.
Anyway, I have no experience in this field, any advice (with examples) would be appreciated.
I would be configuring/building the servers in the UK and shipping to their server farm in the US. Basically I was thinking of using mac minis as they are standardised.
My theory is that the cost of adding more servers is relatively low and I will be able to devote new servers to specific sections that need more grunt.
e.g.: Tomcat servers, file servers, database mirrors etc.
Anyway, I have no experience in this field, any advice (with examples) would be appreciated.
Mac minis use laptop hard disks, which not only means poor performance but also poor reliability, as they just aren't designed for server work. Only do this if you're willing to treat them as disposable, which means very frequent backups and replacement Macs ready to go.
(Why are you shipping them to America?)
posted by cillit bang at 4:04 PM on August 25, 2006
(Why are you shipping them to America?)
posted by cillit bang at 4:04 PM on August 25, 2006
If your servers are hosted in any sort of normal datacenter, they will need to be rack-mounted machines designed to screw into a standard-sized rack. Servers sold for this purpose also have fans, disks, and power supplies to match. Also, you won't need to pay for the audio, 3D graphics, combo drive, wireless, and other components you won't use in the Mac.
posted by mbrubeck at 5:09 PM on August 25, 2006
posted by mbrubeck at 5:09 PM on August 25, 2006
Best answer: This is a cavalcade of bad ideas.
First, if you're colocating, you want the sort of things that the people in the colo centre are accustomed to, as much as possible -- especially since you'll be using remote hands for everything. Remote-hands techs aren't quick on their feet, so it's good to benefit from familiarity, especially if they might need to get inside the box.
Since this sounds like a large commercial web application, you probably also want hardware that has on-site support with a short turnaround, so you can call the vendor, tell them to go to the data centre and fix your stuff. You can't get that from Apple, at least not on Minis.
Servers from server vendors are all standardized. Dell, HP, Compaq, IBM, Sun. There's nothing special about Apple there.
If it's not a large commercial web application you'd be better off just buying a single big machine instead of managing a bunch of little ones. When it comes to scheduling, the kernel is smarter than you are. (Unless the machines are running Windows, in which case one-service-per-server is standard practice.)
And hosting somewhere you're not is very often a bad idea too -- unless it solves a particular problem that is unsolvable any other way (but latency and bandwidth are not those problems). It just means that all of a sudden a strange country's laws apply to your equipment, and that you will have a very hard time doing anything if the company you're dealing with goes sour. Even having to deal with customs when you ship hardware there is a pain in the ass.
Lastly, be absolutely sure you need to colocate your own hardware. My rule of thumb is that unless the hosting vendor is particularly unreliable (and thus you shouldn't be there anyhow), or you already have an established hardware vendor relationship, or you're installing at least a full half-rack of equipment including border networking, you're better off with any of shared, dedicated, managed, or ASP hosting, because with those you're taking advantage of the hosting provider's economy of scale and shifting responsibility for the things that they do daily anyhow over to them instead of taking it on yourself.
This just all sounds like the wrong way to do it. It also sounds like you're trying to come up with an interesting solution, but interesting is rarely better: a cluster is more interesting than a single larger machine but it just means more parts to fail and more OSes to manage; hosting on Macs instead of PCs is more interesting but it means you can't take advantage of everyone being familiar with PCs; hosting in the States instead of in the nearest big city is high-profile but doesn't necessarily get you better connectivity or service, etc.
(I hate listing answering-credentials here, but: i'm a senior Unix sysadmin at a midsize Canadian business telephony vendor, who has been burned by colocating servers in the States when it wasn't necessary, by using desktop Macs for inappropriate commodity-server tasks (at a previous job), and both by splitting across machines when unnecessary and by combining on one machine when splitting was a better bet, all at one time or another. Let my experience help you avoid pain.)
posted by mendel at 5:17 PM on August 25, 2006
First, if you're colocating, you want the sort of things that the people in the colo centre are accustomed to, as much as possible -- especially since you'll be using remote hands for everything. Remote-hands techs aren't quick on their feet, so it's good to benefit from familiarity, especially if they might need to get inside the box.
Since this sounds like a large commercial web application, you probably also want hardware that has on-site support with a short turnaround, so you can call the vendor, tell them to go to the data centre and fix your stuff. You can't get that from Apple, at least not on Minis.
Servers from server vendors are all standardized. Dell, HP, Compaq, IBM, Sun. There's nothing special about Apple there.
If it's not a large commercial web application you'd be better off just buying a single big machine instead of managing a bunch of little ones. When it comes to scheduling, the kernel is smarter than you are. (Unless the machines are running Windows, in which case one-service-per-server is standard practice.)
And hosting somewhere you're not is very often a bad idea too -- unless it solves a particular problem that is unsolvable any other way (but latency and bandwidth are not those problems). It just means that all of a sudden a strange country's laws apply to your equipment, and that you will have a very hard time doing anything if the company you're dealing with goes sour. Even having to deal with customs when you ship hardware there is a pain in the ass.
Lastly, be absolutely sure you need to colocate your own hardware. My rule of thumb is that unless the hosting vendor is particularly unreliable (and thus you shouldn't be there anyhow), or you already have an established hardware vendor relationship, or you're installing at least a full half-rack of equipment including border networking, you're better off with any of shared, dedicated, managed, or ASP hosting, because with those you're taking advantage of the hosting provider's economy of scale and shifting responsibility for the things that they do daily anyhow over to them instead of taking it on yourself.
This just all sounds like the wrong way to do it. It also sounds like you're trying to come up with an interesting solution, but interesting is rarely better: a cluster is more interesting than a single larger machine but it just means more parts to fail and more OSes to manage; hosting on Macs instead of PCs is more interesting but it means you can't take advantage of everyone being familiar with PCs; hosting in the States instead of in the nearest big city is high-profile but doesn't necessarily get you better connectivity or service, etc.
(I hate listing answering-credentials here, but: i'm a senior Unix sysadmin at a midsize Canadian business telephony vendor, who has been burned by colocating servers in the States when it wasn't necessary, by using desktop Macs for inappropriate commodity-server tasks (at a previous job), and both by splitting across machines when unnecessary and by combining on one machine when splitting was a better bet, all at one time or another. Let my experience help you avoid pain.)
posted by mendel at 5:17 PM on August 25, 2006
And having wrote all that, here's a better approach: Tell us what you want to do in general terms ("I have an application that... it uses these technologies... the target audience is... my budget is around... I expect this sort of traffic...") and we'll try to turn that into implementation suggestions.
posted by mendel at 5:20 PM on August 25, 2006
posted by mendel at 5:20 PM on August 25, 2006
Use XServes, if you want to use Apple. I wouldn't, though. Personally, if you're doing a lot of parallelism and/or database work and/or storage, Solaris is an excellent choice, & Sun hardware isn't a huge amount more expensive than x86 hardware. Sun will let you demo their new SunFire systems, which you may want to do.
Other than that, if you're not into watching your servers very closely, buy from a well-known vendor that will support them (ie IBM or HP - I'd stay away from Dell - our department uses them and even for servers, their customer service was shit).
Definitely listen to mendel - I'll put in a nod to all of his advice, although I haven't been through the same experiences.
posted by devilsbrigade at 9:31 PM on August 25, 2006
Other than that, if you're not into watching your servers very closely, buy from a well-known vendor that will support them (ie IBM or HP - I'd stay away from Dell - our department uses them and even for servers, their customer service was shit).
Definitely listen to mendel - I'll put in a nod to all of his advice, although I haven't been through the same experiences.
posted by devilsbrigade at 9:31 PM on August 25, 2006
Someone better tell the folks at Mac Mini Colo that the Mac Minis are terrible as colocated machines. :)
mendel is right - we really need more specifications on what you're trying to do. Are you confusing "distributed" with "load balanced" by any chance? How much traffic? Expected load? Budget?
Whatever you do, I would strongly advise against the "build your own" route. It's usually not cheaper in the long run.
Depending on the database type, Sun is a great choice. Unless it's MySQL, in which case Linux will work just fine. You can cluster MySQL boxes easy enough.
I have always had excellent luck with Dell. Bad BAD luck with HP. Your mileage may vary.
(credentials? IT/sysadmin contracts or full time jobs at: Genentech, Microsoft, Apple, NASA, and Netscape/AOL. As a side gig, owns own web hosting company. Now pretty much hates computers. heh.)
"on-site support with a short turnaround, so you can call the vendor, tell them to go to the data centre and fix your stuff" - make sure your colo will even let them in. That's a problem I've run into before. The colo people absolutely refused to let anybody in unless we were there with them. Strict security measures n' all.
posted by drstein at 10:50 PM on August 25, 2006
mendel is right - we really need more specifications on what you're trying to do. Are you confusing "distributed" with "load balanced" by any chance? How much traffic? Expected load? Budget?
Whatever you do, I would strongly advise against the "build your own" route. It's usually not cheaper in the long run.
Depending on the database type, Sun is a great choice. Unless it's MySQL, in which case Linux will work just fine. You can cluster MySQL boxes easy enough.
I have always had excellent luck with Dell. Bad BAD luck with HP. Your mileage may vary.
(credentials? IT/sysadmin contracts or full time jobs at: Genentech, Microsoft, Apple, NASA, and Netscape/AOL. As a side gig, owns own web hosting company. Now pretty much hates computers. heh.)
"on-site support with a short turnaround, so you can call the vendor, tell them to go to the data centre and fix your stuff" - make sure your colo will even let them in. That's a problem I've run into before. The colo people absolutely refused to let anybody in unless we were there with them. Strict security measures n' all.
posted by drstein at 10:50 PM on August 25, 2006
Response by poster: Okay, the general idea I get from all of you guys is that I really shouldn't go down the mini path as it is just a fad. Which is true, somehow I got it in my head that it would be good to work in house (literally) on these boxes as they are small and non noisy. But I can see that in production they would not handle the demand (or offer the reliability).
Company Background: My website is a service oriented system that provides a CPU/DB intensive product that customers use very often. It also contains a work distribution system that sends out parcels of work to employees who are all connected via a desktop application. Basically with my current server environment (ev1servers standalone 1.6G server) I can see that under high load the server cannot handle the stress.. Mainly because it is sharing with DB/fs/Tomcat and all of them want CPU/io. We are soon to add a new product and I am absolutely sure that this new product will kill the server as it is going to be rediculously popular (in its niche) and thus add even more load to the db/cpu/memory. Also, I have already implemented caching and indexing, but, it is simply the sheer volume of requests that come that cause the high load (there are a few more tweaks I can add but they will not solve the issue).
I have seen a few mentions of Sun boxes, and I have to admit that I was interested in the multi CPU sun boxes. My only concern was that by investing in one powerful sun box I lose the advantage of server redundancy and thus cannot gurantee 100% uptime for my website. This is where I thought the mac minis would help, as I could add redundant servers for all areas.
Will having the db, tomcat AND httpd on the one box limit the number of concurrent users that I have (remembering that for my company DB use is almost guaranteed on a per request basis). Has anyone been in a similar situation, and what was done to ensure 99.9999% uptime?
Thanks everyone for your feedback!
posted by viiviiviivii at 2:30 AM on August 26, 2006
Company Background: My website is a service oriented system that provides a CPU/DB intensive product that customers use very often. It also contains a work distribution system that sends out parcels of work to employees who are all connected via a desktop application. Basically with my current server environment (ev1servers standalone 1.6G server) I can see that under high load the server cannot handle the stress.. Mainly because it is sharing with DB/fs/Tomcat and all of them want CPU/io. We are soon to add a new product and I am absolutely sure that this new product will kill the server as it is going to be rediculously popular (in its niche) and thus add even more load to the db/cpu/memory. Also, I have already implemented caching and indexing, but, it is simply the sheer volume of requests that come that cause the high load (there are a few more tweaks I can add but they will not solve the issue).
I have seen a few mentions of Sun boxes, and I have to admit that I was interested in the multi CPU sun boxes. My only concern was that by investing in one powerful sun box I lose the advantage of server redundancy and thus cannot gurantee 100% uptime for my website. This is where I thought the mac minis would help, as I could add redundant servers for all areas.
Will having the db, tomcat AND httpd on the one box limit the number of concurrent users that I have (remembering that for my company DB use is almost guaranteed on a per request basis). Has anyone been in a similar situation, and what was done to ensure 99.9999% uptime?
Thanks everyone for your feedback!
posted by viiviiviivii at 2:30 AM on August 26, 2006
Generally a pair of reliable boxes, each with RAID, will be enough to guarantee excellent stability. At that point, peering and operator error will likely be your weak points. If there's really a lot of DB access, separating the DB out is probably a good idea. If its Oracle, separating the DB out is an excellent idea. Tomcat & Apache can easily be on the same box, as can email/etc as long as you're not talking massive quantities.
Sun hardware is, in general, quite stable and scalable. If these boxes are going to be remote, having a couple very reliable boxes will be better than a shitload of non-reliable boxes.
The other thing to keep in mind is that if you want insanely high uptime, you need managed servers, or you need to be right with your servers if something happens. If its really *that* important to you, I'd VERY strongly suggest going with a company like rackspace.com.
Unless you have someone at the datacenter you're coloing out that's going to watch your servers & help you out at a moments notice, I really think you should go managed dedicated.
posted by devilsbrigade at 3:29 AM on August 26, 2006
Sun hardware is, in general, quite stable and scalable. If these boxes are going to be remote, having a couple very reliable boxes will be better than a shitload of non-reliable boxes.
The other thing to keep in mind is that if you want insanely high uptime, you need managed servers, or you need to be right with your servers if something happens. If its really *that* important to you, I'd VERY strongly suggest going with a company like rackspace.com.
Unless you have someone at the datacenter you're coloing out that's going to watch your servers & help you out at a moments notice, I really think you should go managed dedicated.
posted by devilsbrigade at 3:29 AM on August 26, 2006
thus cannot gurantee 100% uptime
Whoa whoa whoa whoa whoa. Here's our major disconnect: you can't guarantee 100% uptime, period. The usual off-the-cuff way about talking about uptime is in "nines" -- "three nines" is 99.9% uptime, "four nines" is 99.99%, and so on. (I see on rereading that you knew this, but I might as well leave it in for passersby.)
The high end of availability outside of mission-critical hardware (stock exchanges, air traffic control, hospitals) is five nines. 99.999% uptime means less than five minutes downtime a year. That's not five minutes unscheduled, that's total. One reboot can take that long, let alone actually doing any repairs while the thing is down.
Six nines? Thirty seconds a year. That's carrier-class. You could barely afford to restart a single application at that point. (But also consider that if six nines is carrier class, that any hosting provider is not going to be trying to provide six nines, and it doesn't matter if your systems are running if their uplink is nowhere to be found.)
There is a direct (and probably exponential) relationship between uptime and cost, so here's where you want to figure out what sort of availability your business requires, because it is foolish to spend money on availability you don't need. So you need to figure out if the difference between 5 minutes and 31 seconds downtime per year is worth thousands or tens of thousands of dollars of hosting and development fees, and if the difference between 5 minutes and an hour (four nines) is worth thousands, and so on. And frankly, if you had needed six nines for business reasons, you'd probably be asking your consulting team, not AskMe!
When the most critical systems I'm directly responsible for are unavailable, no-one who has bought our products can activate them -- but even then, a couple minutes a month of unscheduled downtime and an hour every couple months of scheduled downtime is fine by our customers. Sure, we could make it so we don't have to take that hour, but it would mean putting money into something that we wouldn't get any more money back out of.
I'd say one thing you want to look for in a dedicated/managed hosting provider is scalability: underbuy a bit to begin with, and then when you find that's not enough, be able to put more resources online as you need them. Splitting things up into DB and not-DB does make sense from what you describe, and it's also helpful from the point of view of keeping the database "inside" instead of Internet-connected.
(I'm also a bit surprised that an apache/tomcat/DB setup is IO-bound. Make sure your database is able to keep all of its indexes in memory, and that apache/tomcat/the OS can keep most of the application and static pages/images in cache too. And make sure that IO is application IO, not swapping! If it's write-bound, see if you really need everything it's logging, see if you can get webserver and redo logs on separate spindles, and make sure it's logging async.)
Also, I see some recommendations for RAID in there. Be sure you use RAID for performance as well as redundancy: your scenario is one for RAID 1 or 10, not 5.
Lastly, although my experience might be isolated, I've found rackspace.com particularly incompetent on a number of occasions: once, accidentally deinstalling our (running) server from the top half of a rack in which the tenant of the bottom half was moving, and once after a remote-hands reboot, finding that a wedged Linux server of ours came up fine... with a Windows login screen. "I think you should probably give that other customer a call now." But the kind of thing they provide is right up your alley.
And digressing:
Someone better tell the folks at Mac Mini Colo that the Mac Minis are terrible as colocated machines. :)
I'd be happy to! They're the colo company, not the colo customers. They're concerned about making money from people that want to colo their Mac Minis, not about making sure people who want to colo some kind of server get the most reliable and cost-effective platform. Don't confuse advice and marketing.
posted by mendel at 3:51 PM on August 26, 2006
Whoa whoa whoa whoa whoa. Here's our major disconnect: you can't guarantee 100% uptime, period. The usual off-the-cuff way about talking about uptime is in "nines" -- "three nines" is 99.9% uptime, "four nines" is 99.99%, and so on. (I see on rereading that you knew this, but I might as well leave it in for passersby.)
The high end of availability outside of mission-critical hardware (stock exchanges, air traffic control, hospitals) is five nines. 99.999% uptime means less than five minutes downtime a year. That's not five minutes unscheduled, that's total. One reboot can take that long, let alone actually doing any repairs while the thing is down.
Six nines? Thirty seconds a year. That's carrier-class. You could barely afford to restart a single application at that point. (But also consider that if six nines is carrier class, that any hosting provider is not going to be trying to provide six nines, and it doesn't matter if your systems are running if their uplink is nowhere to be found.)
There is a direct (and probably exponential) relationship between uptime and cost, so here's where you want to figure out what sort of availability your business requires, because it is foolish to spend money on availability you don't need. So you need to figure out if the difference between 5 minutes and 31 seconds downtime per year is worth thousands or tens of thousands of dollars of hosting and development fees, and if the difference between 5 minutes and an hour (four nines) is worth thousands, and so on. And frankly, if you had needed six nines for business reasons, you'd probably be asking your consulting team, not AskMe!
When the most critical systems I'm directly responsible for are unavailable, no-one who has bought our products can activate them -- but even then, a couple minutes a month of unscheduled downtime and an hour every couple months of scheduled downtime is fine by our customers. Sure, we could make it so we don't have to take that hour, but it would mean putting money into something that we wouldn't get any more money back out of.
I'd say one thing you want to look for in a dedicated/managed hosting provider is scalability: underbuy a bit to begin with, and then when you find that's not enough, be able to put more resources online as you need them. Splitting things up into DB and not-DB does make sense from what you describe, and it's also helpful from the point of view of keeping the database "inside" instead of Internet-connected.
(I'm also a bit surprised that an apache/tomcat/DB setup is IO-bound. Make sure your database is able to keep all of its indexes in memory, and that apache/tomcat/the OS can keep most of the application and static pages/images in cache too. And make sure that IO is application IO, not swapping! If it's write-bound, see if you really need everything it's logging, see if you can get webserver and redo logs on separate spindles, and make sure it's logging async.)
Also, I see some recommendations for RAID in there. Be sure you use RAID for performance as well as redundancy: your scenario is one for RAID 1 or 10, not 5.
Lastly, although my experience might be isolated, I've found rackspace.com particularly incompetent on a number of occasions: once, accidentally deinstalling our (running) server from the top half of a rack in which the tenant of the bottom half was moving, and once after a remote-hands reboot, finding that a wedged Linux server of ours came up fine... with a Windows login screen. "I think you should probably give that other customer a call now." But the kind of thing they provide is right up your alley.
And digressing:
Someone better tell the folks at Mac Mini Colo that the Mac Minis are terrible as colocated machines. :)
I'd be happy to! They're the colo company, not the colo customers. They're concerned about making money from people that want to colo their Mac Minis, not about making sure people who want to colo some kind of server get the most reliable and cost-effective platform. Don't confuse advice and marketing.
posted by mendel at 3:51 PM on August 26, 2006
This thread is closed to new comments.
posted by matthewr at 3:55 PM on August 25, 2006 [1 favorite]