Examples of large software systems built with well-known open source components?

May 4, 2008 10:23 PM

Can you think of examples of large or important software systems built with well-known open source components? I could use them in the introduction of a paper that I'm writing.

I'm trying to write an paper on "opportunistic software development." (For what it's worth, I didn't coin the term. There was a recent call for articles from IEEE Software magazine on this topic.) It would help bolster my case if I could give some impressive examples.

Opportunistic software development involves grabbing a bunch of components and writing glue or wrapper code to cobble them together, rather than writing everything from scratch. While few systems have been built purely from scratch-- there were function libraries even in the "old" days-- the claim is that this is occurring on a scale not seen before, due in part the availability of open source code. Intuitively, I think this is true, because it's common practice among the graduate students that I see around me. Do you know of any commercial, production systems that either does something interesting, is big, or important that was built using open source code as components.

Bonus points, if there's an article about the system. Don't worry if there isn't, just point me in the right direction and I'll track down a citation. Thanks, Mefites!

posted by mausburger to computers & internet (25 answers total)

I imagine you already know about Mac OS X.
posted by Tacodog at 10:31 PM on May 4, 2008

This is so obvious I fear there may be something wrong with it: something like half the web servers on the internet are running Linux and Apache.
posted by Class Goat at 10:38 PM on May 4, 2008

Er, how about that thing called The Internet?

TCP
Sendmail
BIND
Apache
Lots of Unix-ish OSes

I wasn't there, but I bet a lot of the other early internet software tools were open source as well (lynx? tin? gopher clients?). I think it's a widely-held belief that the internet would never have gotten off the ground if it hadn't been for open source software.
posted by autojack at 10:52 PM on May 4, 2008

netcraft will tell you what particular website a site is running, and you can see that Google for example runs linux.

Other examples, Chemical bank also runs linux.

Lots of web servers (even MS at one particular time) run linux and apache.

Not wanting to do your paper for you, but netcraft has lots of articles too.
posted by mattoxic at 10:57 PM on May 4, 2008

Are you interested solely in software applications, or would hardware devices running Linux count? If so, a decent amount of cell phones run Linux... And a lot of arcade games run Linux, such as the MegaTouch that seems ubiquitous in bars and the like.
posted by fogster at 11:04 PM on May 4, 2008

Amazon EC2 runs on open-source components, specifically Linux and Xen as a hypervisor for isolation of the OS instances. I think EC2 is a pretty perfect example of an opportunistic product.
posted by GuyZero at 11:19 PM on May 4, 2008

Would you class Google as large or important?
Your more inside sounds as if you know about software, so I am guessing you accidentally made this question too broad, as I can't imagine an engineering(?) student not knowing about all the examples already given.
posted by bystander at 11:19 PM on May 4, 2008

LiveJournal might be an interesting example. They're "LAMP" (Linux / Apache / MySQL / Perl, although the "P" is usually PHP).

But rather than just take the things and use them, they found themselves building on them. They developed memcached to essentially share cached objects amongst many systems. It's gotten used in many places, from the pathetically small (my blog), to the gigantic (Wikipedia, which has something like 300 servers).

So it shows that it's not just taking OpenSource, it fosters a sense of sharing. I don't know if this is what you're looking for, or if it's radically different from your thesis...

Expanding on my previous answer: a Google for "embedded Linux" might turn up things we're not thinking of.
posted by fogster at 11:20 PM on May 4, 2008

cPanel. It's a webhosting control panel. They sell the panel, something they wrote, but it's basically the "glue" that brings together a bunch of OpenSource apps: Apache, MySQL, some mailserver (it's been years--happy ones--since I ran cPanel), BIND... But also smaller stuff like updating Perl (CPAN), configuring SSH, interfacing with hdparm/SMART controls, chkrootkit, etc.
posted by fogster at 11:24 PM on May 4, 2008

Wasn't Hotmail built on FreeBSD and other open source software before Microsoft bought them out? I'm pretty sure it was Apache/Sendmail/MySQL and there was a big hullabaloo trying to move it all over to Windows NT. That's a huge opportunistic point. They pretty much came up with the idea of offering email via the web, and built it on whatever they could get their hands on.
posted by team lowkey at 11:25 PM on May 4, 2008

Bystander, yes, you're right. I think I did make my question too broad. I'm in software engineering, so I'm looking specifically for software. I'm less interested in hardware and systems with software.

Just running Linux as an OS isn't enough. Otherwise, you could argue that just running Windows and developing an app on top of it was opportunistic.

LiveJournal, cPanel, Hotmail and Amazon EC2 are good examples.

At first blush, xAMP doesn't seem like a good example, but I am having a hard time pinning down exactly why. So, it might be a good example after all, so I will give it some though.

Thanks for the suggestions so far. They're great. Keep them coming.
posted by mausburger at 12:13 AM on May 5, 2008

Both Perl and Java are entirely based around the component model. Perl in particular is ridiculous with CPAN -- really, the first thing I do when I think of writing something, is see if 90% of it is done by some Perl module.
posted by effugas at 1:12 AM on May 5, 2008

Huge amounts of bioinformatics work is done with Perl, incidentally.
posted by effugas at 1:12 AM on May 5, 2008

Salon.com's content management system was based around some mod_perl work. Vignette was based around an embedded TCL engine.

In fact, the whole content management world is into one big mess of products built on open source tools; Alfresco is another example, OpenCMS, you name it.

IBM have a bunch of products that are basically proprietary goo smeared over open-source cores: RAD is Eclipse + Stuff, WebSphere went from a proprietary servlet engine to using Tomcat as its core. IHS is Apache in drag. IBM's efforts are actually quite nice examples, because they have a fairly synergistic relationship with the open source world these days.

Countless wireless routers and the like are based around BSD or Linux with a user-friendly GUI.

HotSip is a telephony product Oracle now own, built on JBoss.

The Samba guys have a bunch of information about commercial products built with Samba as a core.
posted by rodgerd at 1:27 AM on May 5, 2008

I think you are going to find better examples in the "Web 2.0" domain than stand-alone desktop applications. There are tons of applications built on Java or Python, such as Azureas and Eclipse, but I don't think this is the kind of gluing you are talking about. I would expect that sites like Amazon, Digg, Delicious, YouTube, Google and their many lesser known cousins are all using some open source pieces in PHP, Apache, mySQL, XML, etc with Python or Perl or some other scripting language as a wrapper.

One interesting example might the World of Warcraft user interface community. If you look at a framework like the incredibly popular Ace2, there are people building library functions, other people calling those libraries and creating their new mod with dependencies on the libraries, other people piggybacking on top of that mod to add some other functionality, and still others who integrate groups of mods into suites for end-user distribution. It's all in LUA and there is no compiler, so it's all effectively open-source as far as I understand.
posted by sophist at 1:43 AM on May 5, 2008

Oracle Application Server uses the Apache Web Server to serve web pages.

A number of email spam detection appliances run Linux and ClamAV, amongst other programs.
posted by splice at 3:06 AM on May 5, 2008

Java is enormous. I work for a financial services company that handles transaction processing as well as credit, collections and a bunch of client services... all written in Java. One component we use everywhere is Hibernate.
posted by Civil_Disobedient at 3:50 AM on May 5, 2008

The development environments of two major FPGA companies -- Altera and Xilinx -- both use the venerable Tcl for their scripting APIs. (Tcl is under a BSD-style license)

My day job company links several BSD-type license libraries into the product, and we also depend on a GPL-licensed toolchain (compiler and linker) to actually build our application. We produce Windows and Linux binaries in this way.
posted by jepler at 5:28 AM on May 5, 2008

Intuitively, I think this is true, because it's common practice among the graduate students that I see around me.

What academe calls "opportunistic software development" has been standard operating procedure forever in the commercial world (well, at least the 15 years I've been working in it), but usually with a combination of closed- and open-source software. You never build what you can buy, since closed-source components are generally much cheaper than the developer time required to recreate them unless you're paying grad-student wages. It's only as more/better open-source tools have become available that academics have been able to take this approach. For example, when I was in grad school ten years ago, a requirement to do a full-text search would have been filled not with one of the many available commercial tools but with a heroic grad-student coding effort; now you'd use Lucene and be done with it.
posted by backupjesus at 5:32 AM on May 5, 2008

Java is enormous

Java isn't open source
posted by mattoxic at 6:46 AM on May 5, 2008

VLC for the Mac is made with a number of OSS libraries.

Miro is similarly made up of various OSS libraries and is OSS itself.
posted by gen at 7:11 AM on May 5, 2008

Thanks for all the suggestions. They have definitely given me food for thought. I hadn't noticed the relationship between the emergence of web applications and the availability of viable open source components.

Backupjesus, we've done interviews with commercial developers and the phenomenon is a little more nuanced than you describe. I completely agree that "opportunistic software development" is an academic concept that has come late to the game. It is true that people have been doing it for ages. What has changed now are both the size of the software being built and the range of choices that are available. Also, we found examples of commercial developers using open source software as a reference example and then re-implementing the code. They wanted to build a proprietary product that would have violated the license agreement, and this re-implementation allowed them to get around that.
posted by mausburger at 9:08 AM on May 5, 2008

Java isn't open source

Oh, really?
posted by zixyer at 9:25 AM on May 5, 2008

Eclipse, but I'd venture that a lot of the germane systems you'd be curious about are internal to the companies implementing them, regardless of license.
posted by rhizome at 10:08 AM on May 5, 2008

Just to close the loop on this, I found a very nice example: the Science Activity Planner on the Mars Rovers. It was written using eight different open source components (Castor, Java Expression Parser, Xerces-J, MySQL, HSQL Database Engine, VRML97, and Skaringa). Plus it's been written up.

Jeffrey S. Norris, "Mission Critical Development with Open Source Software: Lessons Learned," IEEE Software, pp. 42-49, January/February 2004.
posted by mausburger at 11:49 AM on May 22, 2008