State of the art architecture for a high(ish) demand website?
December 13, 2016 12:53 PM Subscribe
What sort of system architecture / technologies should I propose for a medium to high- traffic website, to ensure stability, scaling, and general everything-is-OK-ness?
I'm putting together a proposal to take over a website which is fairly visible and very strategic for its owners. The site doesn't have very high traffic now, but they're buying TV and radio advertising, so it could increase significantly.
What sort of technologies should I propose? AWS? Heroku? CloudFlare? Is Docker still a thing? What's the go-to for scaling a website on command? Is a single PostgreSQL instance enough? What am I forgetting?
I know you're not my sysadmin, but right now I just want a shopping list of the tech buzzwords I should include.
I have experience building and maintaining websites, but am not really an expert in setting up and deploying for high traffic, I've mostly used webfaction or just a single physical Ubuntu server for most of my projects, but am willing to learn more modern techniques.
I'm putting together a proposal to take over a website which is fairly visible and very strategic for its owners. The site doesn't have very high traffic now, but they're buying TV and radio advertising, so it could increase significantly.
What sort of technologies should I propose? AWS? Heroku? CloudFlare? Is Docker still a thing? What's the go-to for scaling a website on command? Is a single PostgreSQL instance enough? What am I forgetting?
I know you're not my sysadmin, but right now I just want a shopping list of the tech buzzwords I should include.
I have experience building and maintaining websites, but am not really an expert in setting up and deploying for high traffic, I've mostly used webfaction or just a single physical Ubuntu server for most of my projects, but am willing to learn more modern techniques.
How much of the content is static (or static enough that it doesn't have to be generated every single view)? Aggressive caching with things like Varnish at the server level and Akamai at the infrastructure level helps tremendously with high demand sites.
What's the nature of the site? An e-commerce solution is very different from a media consumption site is very different than an online magazine.
Are you still in Chile? A solution for customers there might be different than ones in the US.
posted by Candleman at 1:03 PM on December 13, 2016
What's the nature of the site? An e-commerce solution is very different from a media consumption site is very different than an online magazine.
Are you still in Chile? A solution for customers there might be different than ones in the US.
posted by Candleman at 1:03 PM on December 13, 2016
Response by poster: It's a car site, so the actual car models' info is fairly static, but there's a strong user participation / commenting feature so I don't think it could be 100% static.
It's not transactional, there's no sales, etc., you just look at car models, comment on them, and can request a quote.
And yes, I'm still in Chile, as is the website.
posted by signal at 1:09 PM on December 13, 2016
It's not transactional, there's no sales, etc., you just look at car models, comment on them, and can request a quote.
And yes, I'm still in Chile, as is the website.
posted by signal at 1:09 PM on December 13, 2016
Response by poster: How do you handle scaling in AWS? Is there a slider like in Heroku?
posted by signal at 1:10 PM on December 13, 2016
posted by signal at 1:10 PM on December 13, 2016
The question feels a bit cart-before-horse. What's the existing architecture? What are the current pinch points in terms of scaling and performance? Is it the kind of site that gets consistently high traffic, or does it have spikes? If the latter, are the spikes predictable or not? Who's going to be monitoring and maintaining it, both in terms of experience and time? You need to know what it is you're dealing with before you determine architecture and infrastructure options, and if that means pricing in a thorough audit and performance analysis as part of your proposal, so be it.
(There are varieties of all-AWS option like the one that doteatop outlines, but even then you get into questions of latency and redundancy across zones. And Docker is a way of packaging/sandboxing components, so it's not really the same as the other things you've mentioned.)
posted by holgate at 1:10 PM on December 13, 2016 [4 favorites]
(There are varieties of all-AWS option like the one that doteatop outlines, but even then you get into questions of latency and redundancy across zones. And Docker is a way of packaging/sandboxing components, so it's not really the same as the other things you've mentioned.)
posted by holgate at 1:10 PM on December 13, 2016 [4 favorites]
For scaling, you can use AWS autoscaling groups to scale-up and scale-down within certain limits automatically based on monitored metrics. IMO though for a site like this, having a database backend and server application is still a last resort.
Not sure what your preferred tools are, but if you can find a pure-front-end commenting library to handle adding comments on individual car models, and have the "request a quote" form submit to i.e. Firebase or Google Bigtable, you could still theoretically have a static site.
In that situation, I might try to build the website using Hugo, have a database or JSON document somewhere with the matrix of car models and properties, and have a bash script that builds the static website and ships it to S3.
If there's a CMS you prefer to run which might enable your customers to do things like edit content on the site without bothering you, and you want the most modern deployment method (seems like from your description you might), you could have:
CloudFront -> ELB -> CMS nodes -> RDS
Where the CMS nodes are running CoreOS with Ubuntu Docker containers that already have your CMS built and configured in it. This is a configuration you could run in an autoscaling group pretty easily. I'd still try to avoid it if possible though.
posted by doteatop at 2:04 PM on December 13, 2016
Not sure what your preferred tools are, but if you can find a pure-front-end commenting library to handle adding comments on individual car models, and have the "request a quote" form submit to i.e. Firebase or Google Bigtable, you could still theoretically have a static site.
In that situation, I might try to build the website using Hugo, have a database or JSON document somewhere with the matrix of car models and properties, and have a bash script that builds the static website and ships it to S3.
If there's a CMS you prefer to run which might enable your customers to do things like edit content on the site without bothering you, and you want the most modern deployment method (seems like from your description you might), you could have:
CloudFront -> ELB -> CMS nodes -> RDS
Where the CMS nodes are running CoreOS with Ubuntu Docker containers that already have your CMS built and configured in it. This is a configuration you could run in an autoscaling group pretty easily. I'd still try to avoid it if possible though.
posted by doteatop at 2:04 PM on December 13, 2016
Ohhhkay, let's back up a wee bit here - do you know what your targets are for performance, even back of the napkin?
Let's break this down a bit:
What sort of system architecture / technologies should I propose for a medium to high- traffic website, to ensure stability, scaling, and general everything-is-OK-ness?
What is medium to high in this context, what does it need to grow to, and is the traffic seasonal?
What of the three criteria you suggest are you treating as most critical? I'm asking this because you're sourcing advice from metafilter without giving much in the way of guidance and illustrating a fair lack of familiarity with the technologies you might employ to work this to ground. I'm saying this not to be cruel, but because I made my living designing, building and operating these types of infrastructures and answers vary wildly.
What sort of technologies should I propose? AWS? Heroku? CloudFlare? Is Docker still a thing? What's the go-to for scaling a website on command? Is a single PostgreSQL instance enough? What am I forgetting?
It all depends on what you want to do, but broadly here are some guiding principals.
- All of those technologies you mention are still a thing for a variety of reasons and trade offs. You want simplicity which, in my mind, means you want maturity and then you can bridge out in to the new shiny. This means standard 3 tiered architecture, possibly with load balancers. That should give you the ability to swap out tiers for larger systems as necessary.
- There is no, in my experience, slider based scaling, maybe things have advanced since I did this, but I think it just hid the complexity somewhere else and made it hard to troubleshoot.
Start with the basics, what is the CMS platform, what does the CMS platform recommend for a supporting architecture, do they have recommendations for sizing?
Finally, costs. All of this can be ridiculously expensive - cloud is not a panacea for costs, especially if the costs come out in human hours troubleshooting needless complexity or is sized incorrectly.
At the end of the day, start with what you understand, buzzwords don't get you anywhere. You don't need to be netflix, slack, github, uber, etc etc. You are not them. You need simple, predictable, and easy to troubleshoot. Down the path of the cutting edge lies madness and should be used by people with large budgets and ample staff.
posted by iamabot at 9:56 PM on December 13, 2016 [2 favorites]
Let's break this down a bit:
What sort of system architecture / technologies should I propose for a medium to high- traffic website, to ensure stability, scaling, and general everything-is-OK-ness?
What is medium to high in this context, what does it need to grow to, and is the traffic seasonal?
What of the three criteria you suggest are you treating as most critical? I'm asking this because you're sourcing advice from metafilter without giving much in the way of guidance and illustrating a fair lack of familiarity with the technologies you might employ to work this to ground. I'm saying this not to be cruel, but because I made my living designing, building and operating these types of infrastructures and answers vary wildly.
What sort of technologies should I propose? AWS? Heroku? CloudFlare? Is Docker still a thing? What's the go-to for scaling a website on command? Is a single PostgreSQL instance enough? What am I forgetting?
It all depends on what you want to do, but broadly here are some guiding principals.
- All of those technologies you mention are still a thing for a variety of reasons and trade offs. You want simplicity which, in my mind, means you want maturity and then you can bridge out in to the new shiny. This means standard 3 tiered architecture, possibly with load balancers. That should give you the ability to swap out tiers for larger systems as necessary.
- There is no, in my experience, slider based scaling, maybe things have advanced since I did this, but I think it just hid the complexity somewhere else and made it hard to troubleshoot.
Start with the basics, what is the CMS platform, what does the CMS platform recommend for a supporting architecture, do they have recommendations for sizing?
Finally, costs. All of this can be ridiculously expensive - cloud is not a panacea for costs, especially if the costs come out in human hours troubleshooting needless complexity or is sized incorrectly.
At the end of the day, start with what you understand, buzzwords don't get you anywhere. You don't need to be netflix, slack, github, uber, etc etc. You are not them. You need simple, predictable, and easy to troubleshoot. Down the path of the cutting edge lies madness and should be used by people with large budgets and ample staff.
posted by iamabot at 9:56 PM on December 13, 2016 [2 favorites]
[Down the path of the cutting edge lies madness and should be used by people with large budgets and ample staff] ->This!
A good host such as Bluehost and Wordpress may work for you - you can purchase themes from Themeforest or other reputed places that are photo-library oriented and have robust comment-plugins available.
As to quotes, not sure where that comes from currently, but can be coded as a simple SQL query with a Javascript enabled front-end.
posted by theobserver at 2:29 AM on December 14, 2016
A good host such as Bluehost and Wordpress may work for you - you can purchase themes from Themeforest or other reputed places that are photo-library oriented and have robust comment-plugins available.
As to quotes, not sure where that comes from currently, but can be coded as a simple SQL query with a Javascript enabled front-end.
posted by theobserver at 2:29 AM on December 14, 2016
« Older Looking for help with purchasing a safety razor. | Looking for Amiga music tracker that produced an... Newer »
This thread is closed to new comments.
For static content (isn't it all static content?), use CloudFront, and if possible, keep the static content in S3. If the content is entirely static, you can do *all* of it just using this first sentence and it will cost you from nothing to almost nothing.
For dynamic content/servers, try having 2-3 Ubuntu servers (across two availability zones) behind an ELB.
For Postgres, if you need a DB, just use RDS IMO. AWS will take care of configuring it and keeping it up. Much happier than managing your own.
posted by doteatop at 12:59 PM on December 13, 2016