Concern about web page photo gallery and impact on server load

April 24, 2014 4:41 PM

I'm setting up a new set of image gallery web pages for a site recently redesigned with Twitter Bootstrap framework. Based on past experience I'm concerned about possible server load issues, but since this is new tech now maybe I'm just paranoid? Full explanation inside...

Web site is for a yearly convention/social event that people come to from around the world. Lasts over a weekend, and each day we post a gallery of pictures from the previous day, anywhere from 200 to 800 pictures per day. The previous version of the site, created around 5 or 6 years ago, used Gallery2. The first year we used it, we allowed the "slideshow" option, and on the last day of the event, we had so many server hits using the slideshow that our webhost shut us down for a couple hours. Our most-viewed day of the year, and our site was offline. I can't have that happen again, so I never again allowed the slideshow view in the Gallery2 software.

In January I redesigned the site using Bootstrap to allow for mobile responsiveness (our #1 browser is iPhone) and it's been extremely well received. Only thing I haven't finished yet is the photo galleries. I think I've settled on using this "Bootstrap Image gallery" package. It doesn't auto-create thumbnails and it doesn't auto-paginate, which I would love, but I think I can get around that.

My concern is about how many images to include per page, and whether the slideshow view could possibly cause the same server load issues we had before. It would be awesome if I could just have one page of images per day, but when there are 600+ pictures, that's a helluva lot of thumbnails. So far I've only tested with a page of just under 200 images, and I must say I was pleasantly surprised at how quickly it loaded on my iPhone over LTE (not WiFi). (Our webhost now uses Google Page Speed Optimization tools.)

The full size images range in file size from 60KB to 120KB, with the average probably somewhere around 80-90KB.

Other possible concern - In order to avoid having to manually create the links to all the images, I've come up with a way in PHP to scan a folder and create them automatically. But that means the server is going to have to work to scan the folders every time someone loads the page. I thought maybe I could run that once on the server then copy/paste from the resulting page source, to create a static file, but the page speed optimizations add a piece of code to each image link so I'd have to remove those manually, which defeats the "not manually editing hundreds of links" purpose.

In short, when a few hundred thousand people in one day try to view hundreds of pictures through this Bootstrap slideshow, is this going to make our webhost freak out again, or will this be OK, given the new technology today and that this collection of CSS and scripts is much smaller than the previous one? I'm not fully tech-savvy, this is a side project not my actual job so I'm not as fully up to speed with these kinds of issues as I could be, so that's why I'm reaching out to you folks.

Thanks for any advice, and if I haven't explained things well let me know and I'll try to clarify.

posted by dnash to Computers & Internet (16 answers total)

This is quick and dirty, but two things I'd consider:

Get a batch renaming program and rename the images something predictable like [eventname]_[date]_[number]

Now you don't have to do anything crazy in php; as long as you know the total number of images it's easy to generate links programmatically.

It sounds like your server might have gotten killed by huge amounts of requests for images. I'd consider hosting them somewhere else, like Amazon S3, and linking to them. Your webserver should be serving webpages, not static files. Most big sites have another domain (www.facebookmedia.com or something), but a lot of people use S3 too.
posted by drjimmy11 at 5:01 PM on April 24, 2014

The amount of JS and CSS in a page shouldn't matter, although less files is better than more. It would literally be hard to slow down a website with too much css if you tried. Text is incredibly small compared to media.
posted by drjimmy11 at 5:03 PM on April 24, 2014

Also, I know you mentioned the slideshow doesn't paginate or create thumbs, but both those things are huge. If most people look at a lot of thumbs and only a few big images, you don't want to load all the big images they don't want. You probably know this, but even if you scale them down with css you're still doing the work of loading the whole file.

And if most people don't look at all 600, the ideal solution is to "lazy load" say 20 at a time, and get more as they scroll down or whatever and need more.
posted by drjimmy11 at 5:07 PM on April 24, 2014 [1 favorite]

It depends. There are two things that could cause your host to freak out, and you don't specify which one:

1. CPU Load - Gallery2 shoves images through a PHP script so that its permissions model works correctly, so it can potentially cause more server load than if it just let your webserver serve the images directly (static files almost always take fewer resources to serve than scripts). If all your new solution does is scan the directory for new images and adds them to a list (which you're then caching somewhere so it doesn't have to regenerate every single time, right?), then that's probably going to reduce the server load a fair bit. If you still have your Gallery2 install around you could upload a small set of images to it and your new system and then compare the CPU load with both, if you want to really reassure yourself. If you don't know how to measure CPU load, you might ask your web host for assistance.
2. Bandwidth - It really doesn't matter whether it's Gallery2 or something else serving those images when it comes to bandwidth. They're going to cost the same amount either way. However, it should be easy enough to estimate how much bandwidth will be used by figuring how big an average picture will be (in KB) and then multiply that by the number of pictures in the slideshow and by the number of users you expect to see in the worst case.

All that said, I'd either recommend against providing a slideshow (instead provide a page with thumbnails that link to larger images, and add some script to delay-load the images the user can't see), or if you do, make sure that it doesn't auto-advance without user intervention (so that a user can't just visit it and eat your bandwidth while they ignore it). Unless it's something users are clamoring for, it seems like an unnecessary and potentially costly feature to have.

If you must have an auto-advancing slideshow, I'd recommend making a YouTube video of the pictures and embed that instead. That way you don't pay any of the costs for hosting it aside from the page that embeds the video.
posted by Aleyn at 5:15 PM on April 24, 2014

Response by poster: Aleyn - thank you, you just brought up something I didn't discuss in the post that I think might be the central issue. I'd sort of forgotten that indeed the issue that crashed the server was the auto-advancing slideshow. I turned that off, but a viewer could still pull up a full-size view (not in a "lightbox," but within the Gallery page) and just keep clicking "next." This solution I'm looking at now would be the same thing, as far as I can see - it's not auto-advancing, you just keep clicking "next."

The catch though is the Gallery solution made pretty large thumbnails, and auto-paginated so it would only show 9 at a time. With the large thumbnails it's more likely a user will browse and only view full-size for a smaller number of pictures. This new Bootstrap solution, I have control of the thumbnail size but the larger they are the fewer I can safely put on a page and the more individual pages I will have to manually create. With smaller thumbnails, it's more likely a viewer - especially on a mobile device - will click to full-size images and swipe through them.
posted by dnash at 5:28 PM on April 24, 2014

Oh, regarding this:

Other possible concern - In order to avoid having to manually create the links to all the images, I've come up with a way in PHP to scan a folder and create them automatically. But that means the server is going to have to work to scan the folders every time someone loads the page. I thought maybe I could run that once on the server then copy/paste from the resulting page source, to create a static file, but the page speed optimizations add a piece of code to each image link so I'd have to remove those manually, which defeats the "not manually editing hundreds of links" purpose.

Have you considered running your script on a machine that you control so that it doesn't go through the Google Page Speed Optimization stuff? Then you'd just upload the static result to your webserver, no editing needed.
posted by Aleyn at 5:34 PM on April 24, 2014

Would it be possible to off-load the gallery functionality to a dedicated web service like smugmug? You could customize layout and domain to match your organization, but let their servers do the heavy lifting.
posted by any portmanteau in a storm at 5:40 PM on April 24, 2014

Response by poster: Get a batch renaming program and rename the images something predictable like [eventname]_[date]_[number]

Now you don't have to do anything crazy in php; as long as you know the total number of images it's easy to generate links programmatically.

drjimmy11 - Thanks. I use Adobe Bridge & Photoshop. Bridge renames. I'm still tweaking a Photoshop action to create the thumbnails.

The solution I'm looking at needs this:

<a href="imagefilename.jpg"><img src="thumbnailname.jpg"></a>

Like I said, I'm not fluent in PHP and JavaScript, but I found a PHP way to generate a list of filenames that are in a certain directory. So I'm going to name the thumbnail files exactly the same with a small prefex ("imagefilename.jpg" and "thumb_imagefilename.jpg") so that the PHP code can use the file name it finds in both places.
posted by dnash at 5:42 PM on April 24, 2014

Response by poster: Have you considered running your script on a machine that you control so that it doesn't go through the Google Page Speed Optimization stuff? Then you'd just upload the static result to your webserver, no editing needed.

Y'know, it hadn't occurred to me but I just might be able to do this. I own another domain myself, on another host - I'm not sure if I can run PHP on it but maybe I can.

It might be too much work to do during the actual event weekend, because it would mean uploading hundreds of files to TWO servers not one, via a hotel Internet connection (and those are never the best), but if I do it now for previous years (yes, we keep all previous years' galleries up) then the only new "dynamic" pages will be the brand new ones, and I can turn them static later.
posted by dnash at 5:46 PM on April 24, 2014

Would it be possible to off-load the gallery functionality to a dedicated web service like smugmug?

Yeah, I went into full-on programming mode there, but if it was me I'd put them on flickr or 500px, pay for a pro account as needed, and let people have at it, honestly.
posted by drjimmy11 at 5:57 PM on April 24, 2014

Well, these days, you should have a fair number of options for webhosting that won't shut your site down competely because of a traffic spike without costing an arm and a leg. There are also options like offloading the images to a CDN, like AWS Coudfront, or Cloudfare.
posted by Good Brain at 6:00 PM on April 24, 2014

If your Apache server is sending large amounts of text (e.g., CSS or JS files), HTML, JSON or XML, you can set up mod_deflate to "deflate" or compress those data types before sending a response to the browser. It won't do much for web-targeted images or music (which are usually already highly compressed) but it'll definitely reduce your overhead with anything highly-compressable, and just about anything text-based can be shrunken a lot (unless it is a random stream of characters).
posted by Blazecock Pileon at 6:03 PM on April 24, 2014

Response by poster: So, tricky thing about offloading the images elsewhere... This event involves some alternative lifestyle communities and while the pictures involved are at most "PG" rated (a bare male butt at most) - there is potential for something like Flickr to object to the images. Just best to keep them in-house.
posted by dnash at 6:36 PM on April 24, 2014

Ooof, let me see if I have this right:

• Each day of convention: hundreds of photos are taken and somehow handed to you (a few photog's SD cards? Some upload tool?)
• End of day: you download these hundreds of photos to a laptop running Bridge and PS.
• PS chugs through and thumbs hundreds of photos.
• You stick those back up on your server with crappy hotel wifi.
• Per request, your server indexes that that folder of thumbs and orgs.
• Gallery is going to download ALL thumbs (no pagination).
• User is going to browse around and download a few, or all hundreds of images.
• User gets distracted by something, comes back. Folders (which have not changed) are indexed again and ALL thumbs downloaded AGAIN.
• Repeat for next day, and next thousand+ users.

1. You need to make sure your server is telling browsers to cache those thumbs.
2. You need to paginate, infinite-scroll, something!
3. You should make that index static, I would suspect this alone will dramatically reduce server load.
4. Make sure your host has some kind of flexible bandwidth rating, where they bump you up a tier instead of throttling. AWS is well regarded for this.

I think default Bootstrap is bloaty, but if you take the time to do a customized package (or use the LESS/SASS version) and remove the modules you're not using, you can thin it out. You can also use a lint tool if you've hacked on your default Bootstrap extensively. Might as well minify it too.

I own another domain myself, on another host - I'm not sure if I can run PHP on it but maybe I can.

Stop your crazy train. What about a local PHP server?
posted by fontophilic at 6:19 AM on April 25, 2014

Response by poster: fontophilic:

Hehe. Very close! Photographer takes pics, picks the good ones, resizes them down to around 300x500px, sends them to me via shared Dropbox. (We used to do USB drives or burned CDs but the coordination of having to find each other is annoying. This way I just look in Dropbox and if files are there, I know I can proceed.)

I use Bridge/PS to rename and watermark the pics. I didn't have to do thumbnails before, but I will now. It doesn't really take that long.

You need to make sure your server is telling browsers to cache those thumbs.

While I'm not totally familiar with it, I believe that's something the Page Speed Optimization tools do. We use Dreamhost, and we just turned this Optimization feature on a couple months ago. Even without the image galleries, I immediately noticed a dramatic difference.

I use the default Bootstrap, minimized, and keep customizations in a separate stylesheet. Easier to keep track of changes if/when Bootstrap updates to a new version.

What about a local PHP server?

Like I said, I'm not super-tech savvy. I'm vaguely aware that something like this exists but have never looked into it. I've been using a subdomain for testing new pages. Point me at an example, maybe?
posted by dnash at 8:08 AM on April 25, 2014

For local development I use MAMP, but it's OSX only. I hear good things about the cross-platform XAMPP.

Basically you run a local webserver and test things before pushing them to a server. You view them in your browser at http://localhost:8888/
posted by fontophilic at 1:30 PM on April 25, 2014