Large scale web applications running on Windows
December 17, 2013 9:14 AM Subscribe
As a long time Linux Engineer I enjoy reading about the technologies used by sites like Twitter, Facebook, Etsy, Yahoo (for things like Flickr) and other large sites to build things out to scale. I often lend a hand with Windows customers at work and it recently occurred to me that I've never seen a website or blog posts that relate to scaling on Windows.
In particular I'd be interested in seeing how large MS SQL instances are handled (sharding, masters, slaves, etc.), what technologies are available for app caching (on all of the levels, so memcached, varnish, opcode caching, etc. equivalents), and general large scale administration (I know Puppet Enterprise is available for configuration management, but is it widely used, or is everyone using Powershell, etc.). I'm also sort of curious how Continuous integration/Continuous delivery is done on the Windows side.
I was certain that Microsoft must have posts like this, but Google failed me there. I'm also less interested in them (especially given that they ran Hotmail on a combination of Solaris and FreeBSD for most of the early part of its life), and more interested in others. Someone has to be running a large scale web app (let's define large as hundreds of VMs in multiple geo-distant datacenters) on Windows and talking about it. I'd like to read about those people, and now that I've mentioned it, I'd prefer to read about it, not watch videos.
In particular I'd be interested in seeing how large MS SQL instances are handled (sharding, masters, slaves, etc.), what technologies are available for app caching (on all of the levels, so memcached, varnish, opcode caching, etc. equivalents), and general large scale administration (I know Puppet Enterprise is available for configuration management, but is it widely used, or is everyone using Powershell, etc.). I'm also sort of curious how Continuous integration/Continuous delivery is done on the Windows side.
I was certain that Microsoft must have posts like this, but Google failed me there. I'm also less interested in them (especially given that they ran Hotmail on a combination of Solaris and FreeBSD for most of the early part of its life), and more interested in others. Someone has to be running a large scale web app (let's define large as hundreds of VMs in multiple geo-distant datacenters) on Windows and talking about it. I'd like to read about those people, and now that I've mentioned it, I'd prefer to read about it, not watch videos.
MySpace used to be the biggest MS based site, by far. These days, I have no idea, probably Bing and not too many others.
There is another reason there is a lot less Windows scalability porn because most people who choose Microsoft do so because they find the MS ecosystem valuable. That means if you focus on features microsoft provides. MS-SQL has its own high availability features, it has master slave replication. It probably has a "sharding" solution, but probably using a more traditional term, like "partitioning."
Opcode caching is pretty much a non issue. ASP.net is processed into C#, then c# to common intermediate language, which is then turned into a binary. The microsoft stack deals with making this stuff happen automatically.
For configuration management, I'd start by looking at System Management Server, but also understanding that msi installers and ActiveDirectory are part of the story.
I am not saying these things are all suitable for "webscale" deployments, but they are often starting points. Of course another part of the puzzle for large scale deployments is Azure. Again, not everyone is using it, but it is going to embody Microsoft's latest thinking on how people should be doing large scale deployments of MS servers.
posted by Good Brain at 9:48 AM on December 17, 2013
There is another reason there is a lot less Windows scalability porn because most people who choose Microsoft do so because they find the MS ecosystem valuable. That means if you focus on features microsoft provides. MS-SQL has its own high availability features, it has master slave replication. It probably has a "sharding" solution, but probably using a more traditional term, like "partitioning."
Opcode caching is pretty much a non issue. ASP.net is processed into C#, then c# to common intermediate language, which is then turned into a binary. The microsoft stack deals with making this stuff happen automatically.
For configuration management, I'd start by looking at System Management Server, but also understanding that msi installers and ActiveDirectory are part of the story.
I am not saying these things are all suitable for "webscale" deployments, but they are often starting points. Of course another part of the puzzle for large scale deployments is Azure. Again, not everyone is using it, but it is going to embody Microsoft's latest thinking on how people should be doing large scale deployments of MS servers.
posted by Good Brain at 9:48 AM on December 17, 2013
There's a few decent articles about Plenty of Fish, which has not a huge amount of hardware, but an awful lot of hits. Here's one.
posted by ambrosen at 10:45 AM on December 17, 2013
posted by ambrosen at 10:45 AM on December 17, 2013
Had a minute to find a few more posts about stackexchange:
Designing for scalability
Network Configuration
Not particularly full of database craziness, but it's all I've got.
posted by jsturgill at 2:21 PM on December 17, 2013
Designing for scalability
Network Configuration
Not particularly full of database craziness, but it's all I've got.
posted by jsturgill at 2:21 PM on December 17, 2013
I'd disagree with you, I've read quite a few Microsoft focused articles on High Scalability.
But it's a feed, and they're rarer than Linux articles.
In fact just Google "high scalability windows" and you'll find articles, it might take some digging.
I'd also recommend InfoQ, it puts on software conferences, but publishes very high quality articles, occasionally focused on scalability.
I know we run a large shop, primarily MS, but some Linux (Couchbase, RabbitMQ, et al) doing frequent releases (~150 yr), we use Chef, not Puppet for environment configuration and management. At the end of the day, design your applications to scale linearly. Spend some time building your continous deployment channel. Focus on messaging, what should I cache, where should I cache, where are my bottle necks. All services should be idempotent and redudant.
posted by patrickje at 2:56 PM on December 17, 2013
But it's a feed, and they're rarer than Linux articles.
In fact just Google "high scalability windows" and you'll find articles, it might take some digging.
I'd also recommend InfoQ, it puts on software conferences, but publishes very high quality articles, occasionally focused on scalability.
I know we run a large shop, primarily MS, but some Linux (Couchbase, RabbitMQ, et al) doing frequent releases (~150 yr), we use Chef, not Puppet for environment configuration and management. At the end of the day, design your applications to scale linearly. Spend some time building your continous deployment channel. Focus on messaging, what should I cache, where should I cache, where are my bottle necks. All services should be idempotent and redudant.
posted by patrickje at 2:56 PM on December 17, 2013
http://www.ancestry.com/ is a windows shop, they show up to alot of the meetups around Austin, and I even met some of them at re:Invent last year.
They have a few publicly available talks/posts about their arch.
posted by anthroprose at 7:33 PM on December 17, 2013
They have a few publicly available talks/posts about their arch.
posted by anthroprose at 7:33 PM on December 17, 2013
This thread is closed to new comments.
posted by jsturgill at 9:17 AM on December 17, 2013