How does one build a supercomputer?
February 24, 2011 6:09 AM Subscribe
Computer Theory Filter: In short, what are the major reasons/roadblocks stopping someone from gathering computer processors and parts that are going to be recycled etc. and building a larger and larger supercomputer? (more inside)
I apologize in advance if this is a completely stupid question but this whole Watson thing got me back on an old question I had.
It would seem to me that all of the computers being recycled or thrown away could somehow be stripped down and connected together to become a bigger and bigger computer. I understand that no hardware exists to simply plug in new processors but could this be developed? I imagine it would also be difficult to have everything "talk" to each other but how do they do this with distributed computing now?
Basically feel free to tell me why this does or does not work but keep it relatively simple as I obviously know nothing about computers.
I apologize in advance if this is a completely stupid question but this whole Watson thing got me back on an old question I had.
It would seem to me that all of the computers being recycled or thrown away could somehow be stripped down and connected together to become a bigger and bigger computer. I understand that no hardware exists to simply plug in new processors but could this be developed? I imagine it would also be difficult to have everything "talk" to each other but how do they do this with distributed computing now?
Basically feel free to tell me why this does or does not work but keep it relatively simple as I obviously know nothing about computers.
Interoperability. CPUs, memory, and other components are not built identically; they have different instruction sets, capabilities, and requirements. Much of the effort in systems programming goes into dealing with the wrinkles of particular hardware; it's impossible to write a completely generic underlying layer to deal with arbitrary hardware – at least at the level that a "supercomputer" would require. (This is exactly the approach that virtual machines like the JVM, .NET and the Python VM take, but they're used at application level, not the system level.)
About the closest you can get is the Beowulf Cluster, but note that the devices are identical.
posted by sonic meat machine at 6:24 AM on February 24, 2011 [1 favorite]
About the closest you can get is the Beowulf Cluster, but note that the devices are identical.
posted by sonic meat machine at 6:24 AM on February 24, 2011 [1 favorite]
A lot of "supercomputer" stuff these days does refer to cluster computing, for example universities tend to have clusters these days.
The reason why people don't do it from old things is that it's cheaper and easier to buy new parts that match. You get the benefits of interchangable parts from new stuff in a way that you don't get with old parts.
Also, quality is still an improvement over pure quantity. Imagine building a bookcase with fifty of your closest friends...and you only have two screwdrivers. The bookcase isn't going to go together any faster than if you only had three friends, but if one guy is extremely strong then that would still make it faster. For a cluster, the problem you're solving needs to optimize well for parallel computing, meaning that many computers need to be able to work on it at once--without waiting for results from the other computers in the cluster. Works well for some problems, doesn't work so well for others.
posted by anaelith at 6:27 AM on February 24, 2011 [2 favorites]
The reason why people don't do it from old things is that it's cheaper and easier to buy new parts that match. You get the benefits of interchangable parts from new stuff in a way that you don't get with old parts.
Also, quality is still an improvement over pure quantity. Imagine building a bookcase with fifty of your closest friends...and you only have two screwdrivers. The bookcase isn't going to go together any faster than if you only had three friends, but if one guy is extremely strong then that would still make it faster. For a cluster, the problem you're solving needs to optimize well for parallel computing, meaning that many computers need to be able to work on it at once--without waiting for results from the other computers in the cluster. Works well for some problems, doesn't work so well for others.
posted by anaelith at 6:27 AM on February 24, 2011 [2 favorites]
Supercomputers have specially designed processors not found in consumer goods.
Not so much. Modern supercomputers are huge clusters of commodity parts. Take Tianhe-I, the Chinese supercomputer which is currently the most powerful in the world. It runs on thousands of Intel Xeon X5670 and Nvidia M2050 GPUs. Commodity hardware, specialized software.
posted by sonic meat machine at 6:30 AM on February 24, 2011 [1 favorite]
Not so much. Modern supercomputers are huge clusters of commodity parts. Take Tianhe-I, the Chinese supercomputer which is currently the most powerful in the world. It runs on thousands of Intel Xeon X5670 and Nvidia M2050 GPUs. Commodity hardware, specialized software.
posted by sonic meat machine at 6:30 AM on February 24, 2011 [1 favorite]
Anaelith has the answer: many problems in computing aren't easily parallelized.
If you have a car that only goes 100 mph, you can add as many cars as you want and you still can't go faster than 100 mph.
And then there are economics of scale- you could put 1000 old pentium III machines together and probably still not get the performance (parallel or not) of a $1000 computer off of Dell's website.
posted by gjc at 6:34 AM on February 24, 2011 [1 favorite]
If you have a car that only goes 100 mph, you can add as many cars as you want and you still can't go faster than 100 mph.
And then there are economics of scale- you could put 1000 old pentium III machines together and probably still not get the performance (parallel or not) of a $1000 computer off of Dell's website.
posted by gjc at 6:34 AM on February 24, 2011 [1 favorite]
Not to derail but how many consumer goods run on Intel Xeon chips? Not many I would suspect. I think the OP's question pertains to the question of the processors found in your average consumer good, which suggests a relatively low end processor.
posted by dfriedman at 6:35 AM on February 24, 2011
posted by dfriedman at 6:35 AM on February 24, 2011
Also, hardware is improving quickly enough that you'd need a large number of outdated CPUs to match the processing power of a current one; it wouldn't be cost- or energy-efficient to bother.
posted by ook at 6:35 AM on February 24, 2011
posted by ook at 6:35 AM on February 24, 2011
You probably are thinking: if one processor can compute X amount (in 1 second), 2 processors can compute 2X; thus all the recycled processors in the world PLUS currently used processors must be able to generate more computation than just the currently used processors.
Unfortunately, computation is complex; and they don't add up linearly as you thought. Many problems in computer science can not be broken down to smaller, independent problems. This is the challenge of parallel computing: sub-problems are dependent on each other. Multiplication is a problem that can be parallelize: 1234 * 6789 can be broken down to (1234)*6*10^3 + (1234)*7*10^2 + (1234)*8*10 + (1234)*9; each of which is a smaller problem and all can be done in parallel and recombined. Calculating PI is not easily parallelized, because the next digit of PI depends on the previous digit of PI; so if you set 100 computer, each responsible to calculate one digit of PI, they will all have to wait on each other; and the 100th computer can only start running after all 99 computers before it has finished.
So, you can see that in the case of calculating PI, having more slow computer doesn't help; in fact, it hurts. You'd rather throw away all 99 old computer, and buy one fast one, then compute PI one digit at a time; instead of daisy chain slow computers together. Calculating PI is among the benchmark people use to assess supercomputer performance.
Notice that supercomputer benchmarking is itself a contentious subject. How "powerful" a supercomputer is depends a lot of many different factors, not just number_of_CPU*speed. Most CPUs are data-starved; they spend most of their time waiting for data. This can either be because of data-dependency, or I/O access, or energy limitation, etc. Supercomputers are tuned for specific problem. Depends on granularity of the problem that one make the trade-off between favoring CPU speed or I/O speed.
posted by curiousZ at 6:36 AM on February 24, 2011 [6 favorites]
Unfortunately, computation is complex; and they don't add up linearly as you thought. Many problems in computer science can not be broken down to smaller, independent problems. This is the challenge of parallel computing: sub-problems are dependent on each other. Multiplication is a problem that can be parallelize: 1234 * 6789 can be broken down to (1234)*6*10^3 + (1234)*7*10^2 + (1234)*8*10 + (1234)*9; each of which is a smaller problem and all can be done in parallel and recombined. Calculating PI is not easily parallelized, because the next digit of PI depends on the previous digit of PI; so if you set 100 computer, each responsible to calculate one digit of PI, they will all have to wait on each other; and the 100th computer can only start running after all 99 computers before it has finished.
So, you can see that in the case of calculating PI, having more slow computer doesn't help; in fact, it hurts. You'd rather throw away all 99 old computer, and buy one fast one, then compute PI one digit at a time; instead of daisy chain slow computers together. Calculating PI is among the benchmark people use to assess supercomputer performance.
Notice that supercomputer benchmarking is itself a contentious subject. How "powerful" a supercomputer is depends a lot of many different factors, not just number_of_CPU*speed. Most CPUs are data-starved; they spend most of their time waiting for data. This can either be because of data-dependency, or I/O access, or energy limitation, etc. Supercomputers are tuned for specific problem. Depends on granularity of the problem that one make the trade-off between favoring CPU speed or I/O speed.
posted by curiousZ at 6:36 AM on February 24, 2011 [6 favorites]
I was going to suggest going the route of a Beowulf Cluster, but as the person above stated, those devices are identical. what you would need would be to cluster together computers built with all of these old processors and divvy out the jobs based on what the computers can handle, or just divvy out the jobs with a due date and if it's not done divvy it out again (see BOINC).
A novel idea but I think there are too many issues with the old and slow parts. It might be easier with parts currently being produced, but to have a 2.6 GHz processor going against a 686 is just not fair.
posted by zombieApoc at 6:37 AM on February 24, 2011
A novel idea but I think there are too many issues with the old and slow parts. It might be easier with parts currently being produced, but to have a 2.6 GHz processor going against a 686 is just not fair.
posted by zombieApoc at 6:37 AM on February 24, 2011
Not to derail but how many consumer goods run on Intel Xeon chips? Not many I would suspect. I think the OP's question pertains to the question of the processors found in your average consumer good, which suggests a relatively low end processor.
I'm posting this from a workstation with a Xeon processor; it cost less than $3,000. Not "low-end," per se, but certainly not out of the range of "consumer." Supercomputers used to have specialized architectures; today they just use high-end versions of the same architectures used in my workstation, my laptop, or my PC at home.
posted by sonic meat machine at 6:40 AM on February 24, 2011
I'm posting this from a workstation with a Xeon processor; it cost less than $3,000. Not "low-end," per se, but certainly not out of the range of "consumer." Supercomputers used to have specialized architectures; today they just use high-end versions of the same architectures used in my workstation, my laptop, or my PC at home.
posted by sonic meat machine at 6:40 AM on February 24, 2011
Calculating PI is not easily parallelized, because the next digit of PI depends on the previous digit of PI
Side note: this is not true if you work in hexadecimal, due to an amazing result known as the Bailey-Borwein-Plouffe formula.
posted by Electric Dragon at 6:42 AM on February 24, 2011 [4 favorites]
Side note: this is not true if you work in hexadecimal, due to an amazing result known as the Bailey-Borwein-Plouffe formula.
posted by Electric Dragon at 6:42 AM on February 24, 2011 [4 favorites]
Response by poster: Thanks for all the answers so far this is exactly what I suspected / was looking for.
I guess I was just thinking that with all of the computers being thrown out it would seem that if someone could just build some standard hardware and software that others could just "plug and play" processors into then you could just scavenge for similar chips and put them in.
Then for problems where distributed computing is used (where a parsing of the question isnt an issue) this in house "supercomputer" could be used.
posted by Busmick at 6:51 AM on February 24, 2011
I guess I was just thinking that with all of the computers being thrown out it would seem that if someone could just build some standard hardware and software that others could just "plug and play" processors into then you could just scavenge for similar chips and put them in.
Then for problems where distributed computing is used (where a parsing of the question isnt an issue) this in house "supercomputer" could be used.
posted by Busmick at 6:51 AM on February 24, 2011
sonic meat machine: "Supercomputers have specially designed processors not found in consumer goods.
Not so much. Modern supercomputers are huge clusters of commodity parts. Take Tianhe-I, the Chinese supercomputer which is currently the most powerful in the world. It runs on thousands of Intel Xeon X5670 and Nvidia M2050 GPUs. Commodity hardware, specialized software."
Commodity CPUs and GPUs sure, but you are forgetting the specialized interconnects between the systems, such as InfiniBand and the fact that they shove a whole lot of stuff into tiny rack-mounted cases. Commodity network cards won't cut it except for things like BOINC where high latency is acceptable; weather modeling and such not really.
Another reason that supercomputers aren't built with random throwaway hardware is that the computing power per dollar is higher with modern hardware. The dollars in this case refer to the space you need to house the cluster, the power that that keeps it running, the staff that maintain it, etc.
posted by mkb at 6:51 AM on February 24, 2011
Not so much. Modern supercomputers are huge clusters of commodity parts. Take Tianhe-I, the Chinese supercomputer which is currently the most powerful in the world. It runs on thousands of Intel Xeon X5670 and Nvidia M2050 GPUs. Commodity hardware, specialized software."
Commodity CPUs and GPUs sure, but you are forgetting the specialized interconnects between the systems, such as InfiniBand and the fact that they shove a whole lot of stuff into tiny rack-mounted cases. Commodity network cards won't cut it except for things like BOINC where high latency is acceptable; weather modeling and such not really.
Another reason that supercomputers aren't built with random throwaway hardware is that the computing power per dollar is higher with modern hardware. The dollars in this case refer to the space you need to house the cluster, the power that that keeps it running, the staff that maintain it, etc.
posted by mkb at 6:51 AM on February 24, 2011
Commodity CPUs and GPUs sure, but you are forgetting the specialized interconnects between the systems...
Well, I think the original question was mostly about CPUs and GPUs. ("I understand that no hardware exists to simply plug in new processors but could this be developed?") I'm certainly not forgetting interconnections or downplaying their importance (see my first reply), but the shift from specialized architectures to clustered commodity architectures is significant.
posted by sonic meat machine at 7:25 AM on February 24, 2011
Well, I think the original question was mostly about CPUs and GPUs. ("I understand that no hardware exists to simply plug in new processors but could this be developed?") I'm certainly not forgetting interconnections or downplaying their importance (see my first reply), but the shift from specialized architectures to clustered commodity architectures is significant.
posted by sonic meat machine at 7:25 AM on February 24, 2011
It would make more sense to turn a bunch of old computers into a data center, where CPUs don't need to communicate with each other at supercomputer speeds and where CPU performance is not usually the bottleneck. But even then, it wouldn't be cost-effective for two reasons.
1. It would be a nightmare to maintain. In any large system, parts are constantly failing. Diversity in hardware is bad when you need to constantly replace parts. This would lead to the techs just throwing out any malfunctioning machine without even diagnosing the problem.
2. Newer processors are much more energy efficient. Even if your old processors are free, the cost of running and cooling them will quickly make new CPUs a better flops-per-dollar proposition.
posted by qxntpqbbbqxl at 7:31 AM on February 24, 2011 [1 favorite]
1. It would be a nightmare to maintain. In any large system, parts are constantly failing. Diversity in hardware is bad when you need to constantly replace parts. This would lead to the techs just throwing out any malfunctioning machine without even diagnosing the problem.
2. Newer processors are much more energy efficient. Even if your old processors are free, the cost of running and cooling them will quickly make new CPUs a better flops-per-dollar proposition.
posted by qxntpqbbbqxl at 7:31 AM on February 24, 2011 [1 favorite]
Combining a large amount of slow inefficient complex processors would not create a very good super computer. Most super computers are built with simple efficient high-speed processors that can be run in parallel.
That being said, Google does run a large part of its operations on cheap commodity hardware.
posted by blue_beetle at 7:40 AM on February 24, 2011
That being said, Google does run a large part of its operations on cheap commodity hardware.
posted by blue_beetle at 7:40 AM on February 24, 2011
The electrical power constraints make it not worthwhile, among all the answers above. Newer processors are much more power efficient, and it is eventually cheaper to just buy new ones instead of paying for the electricity of lots of old one.
On preview, what qxntpqbbbqxl said.
posted by procrastination at 7:40 AM on February 24, 2011
On preview, what qxntpqbbbqxl said.
posted by procrastination at 7:40 AM on February 24, 2011
One of the big roadblocks, believe it or not, is the speed of light. A small supercomputer is inherently faster than a big super computer because it takes less time for information to propagate across the system when it needs to.
If your hypothetical supercomputer filled a warehouse, speed of light would be a substantial impediment to getting anything done fast.
If your base clock rate is 2.4 GHz, then in one clock cycle light moves about 12.5 centimeters (about 5 inches). To move across a warehouse would take thousands of cycles.
posted by Chocolate Pickle at 7:56 AM on February 24, 2011
If your hypothetical supercomputer filled a warehouse, speed of light would be a substantial impediment to getting anything done fast.
If your base clock rate is 2.4 GHz, then in one clock cycle light moves about 12.5 centimeters (about 5 inches). To move across a warehouse would take thousands of cycles.
posted by Chocolate Pickle at 7:56 AM on February 24, 2011
I guess I was just thinking that with all of the computers being thrown out it would seem that if someone could just build some standard hardware and software that others could just "plug and play" processors into then you could just scavenge for similar chips and put them in.
In addition to the architectural issues raised above, there are efficiency concerns as well. An older processor will use a lot more electricity - generating much more heat - and achieve fewer computations per watt.
posted by Pogo_Fuzzybutt at 8:25 AM on February 24, 2011
In addition to the architectural issues raised above, there are efficiency concerns as well. An older processor will use a lot more electricity - generating much more heat - and achieve fewer computations per watt.
posted by Pogo_Fuzzybutt at 8:25 AM on February 24, 2011
No one has mentioned distributed computing. It is somewhat related to your question in that it takes a bunch of dislike computers and harnesses their spare computing power. In effect it creates a sort of supercomputer via the Internet with spare cycles.
I as going to list some of the cool projects that utilize it, but here is a great list already.
Now get some of those old computers online and let's cure Alzheimers or reach ET!
posted by patrad at 8:28 AM on February 24, 2011
I as going to list some of the cool projects that utilize it, but here is a great list already.
Now get some of those old computers online and let's cure Alzheimers or reach ET!
posted by patrad at 8:28 AM on February 24, 2011
The simple answers are cooling electricity, software, and space, the long versions of these answers has been hashed out, but I think people are underestimating the "software" aspect.
posted by irishcoffee at 8:50 AM on February 24, 2011
posted by irishcoffee at 8:50 AM on February 24, 2011
This has been done. Check out the "stone soupercomputer," a beowolf-style supercomputer that was built out of a variety of discarded desktop PCs at ORNL circa 2000.
they used it for a number of cool things, such as climate modeling.
I don't think anyone else has done this though, as it is low cost in start-up only- maintenance and fiddling costs must be huge.
posted by rockindata at 9:04 AM on February 24, 2011
they used it for a number of cool things, such as climate modeling.
I don't think anyone else has done this though, as it is low cost in start-up only- maintenance and fiddling costs must be huge.
posted by rockindata at 9:04 AM on February 24, 2011
The simple answer is cost. Even though the parts are free, the cost of re-purposing and maintaining the new Frankenputer would far exceed the cost of just buying the most recent hardware. When you try to do anything of scale, things like power consumption and heat output start playing a major role in cost.
From an engineering standpoint it is much preferable to have homogeneous clusters of hardware. If you look at deep blue's racks, notice each server looks the same. It's much more difficult to optimize parallel computation over heterogeneous systems that have varying amounts of memory, storage and cache. Furthermore, as noted elsewhere, not all those parts play nice together, though if you can build discrete machines you could theoretically build something that is heterogeneous above the network layer. But what would you do with it? Meh.
posted by jeffamaphone at 9:28 AM on February 24, 2011
From an engineering standpoint it is much preferable to have homogeneous clusters of hardware. If you look at deep blue's racks, notice each server looks the same. It's much more difficult to optimize parallel computation over heterogeneous systems that have varying amounts of memory, storage and cache. Furthermore, as noted elsewhere, not all those parts play nice together, though if you can build discrete machines you could theoretically build something that is heterogeneous above the network layer. But what would you do with it? Meh.
posted by jeffamaphone at 9:28 AM on February 24, 2011
The majority of the Computing Centre (not the Data Centre) at CERN is composed of commercial off-the-shelf computers networked together.
posted by alby at 10:19 AM on February 24, 2011
posted by alby at 10:19 AM on February 24, 2011
You can totally do it, and people do it all the time! Granted, you'd need machines that could run some sort of compatible operating system (like linux) that supports clustering. I think the main reason your average person doesn't do this is because most problems can't be solved that way. I mean, if you're doing lots of heavy number crunching using some kind of software that parallelizes well, then yeah, a cluster is what you want. But for your everyday computing tasks, it isn't really going to help you.
Hell, most modern machines have multiple CPUs on one chip, and yet most software isn't really written in a way that takes advantage of it.
posted by Afroblanco at 10:42 AM on February 24, 2011
Hell, most modern machines have multiple CPUs on one chip, and yet most software isn't really written in a way that takes advantage of it.
posted by Afroblanco at 10:42 AM on February 24, 2011
I do occasional work on one of the Top100 Supercomputing Clusters. In the long run it's a matter of power more than anything. Your old machines will take 10 cpus, take up 15x the space, require 20x the power, produce 30x the wasted heat requiring 450x the air conditioning of a single new machine. Numbers may be a bit off (not my department) but you'll spend way more money renting the space and keeping it cool and powering your old machines than you would just buying newer more efficient machines to do the same amount of work. (We have just under 3k nodes, 20k cores, mixed Linux 32/64, Solaris x86/sparc)
posted by zengargoyle at 2:34 PM on February 24, 2011
posted by zengargoyle at 2:34 PM on February 24, 2011
It would most definitely be possible for a dedicated hobbyist to go through a pile of discarded computers, evaluate the hardware in each, run some tests, determine whether it works or not, fix/replace the broken components, install an operating system, and install a distributed computing client like SETI@Home or Folding@Home. However, I hope it's apparent that this would be an incredibly labor intensive project -- some of them are going to have busted power supplies, some are going to have no graphics card, some are going to have bad RAM. You have to have a large pile of spare parts and a lot of free time to get anything useful out of trashed hardware. And even if you have this incredible amount of patience, these clunkers are going to be power-hungry and you're not going to be able to put more than 10 or 15 standard desktop computers in a room with standard electrical service and usual cooling. Compare to purpose-made blade servers which sip power and are meant to fit 24 or 36 units at a time into a rack mount case. This is the only way you can get to thousands or tens of thousands of nodes, and even then it requires data centers with expensive cooling and specialized power delivery.
posted by Rhomboid at 4:41 AM on February 25, 2011
posted by Rhomboid at 4:41 AM on February 25, 2011
I used to work in a scientific computing research group, expending vast amounts of compute time on simulations of molecular dynamics and fluid flow. Another university was once trying to get rid of their 96-way SGI Origin2000 to make way for an upgrade. This machine would have cost over a million a few years previously, and they sold it to us for the princely sum of one pound (plus 17p tax) because it had become too costly and too bulky to maintain and they were desperate to get rid of it, and because we were prepared to just rock up and take it away in a van. I'm not sure we really got that much use out of it either in the end: we could get more oomph for less hassle out of newer machines, and keeping a cranky old IRIX box up and running made our sysadmin sad.
The head of our group also made tentative enquiries about acquiring an old 800-way Cray T3E until (IIRC) he discovered that it liked to eat the best part of a megawatt once you factor in aircon and whatnot.
Basically: you can certainly throw lots of processors together in a room and have them do stuff, but it'll be a total beast to maintain, and could easily cost more in power and aircon than you would spend on buying a faster system new.
posted by doop at 7:14 AM on February 26, 2011 [1 favorite]
The head of our group also made tentative enquiries about acquiring an old 800-way Cray T3E until (IIRC) he discovered that it liked to eat the best part of a megawatt once you factor in aircon and whatnot.
Basically: you can certainly throw lots of processors together in a room and have them do stuff, but it'll be a total beast to maintain, and could easily cost more in power and aircon than you would spend on buying a faster system new.
posted by doop at 7:14 AM on February 26, 2011 [1 favorite]
« Older I want to be more stylish. Help? Looking for both... | Fortunatly, it wasn't a tramp stamp Newer »
This thread is closed to new comments.
Supercomputers have specially designed processors not found in consumer goods.
posted by dfriedman at 6:21 AM on February 24, 2011