Breaking PHP scripts into manageable chunks
June 18, 2007 9:05 PM   Subscribe

Strategies to keep PHP timeouts at bay in a shared hosting environment.

I'm a hacktastic PHP "programmer" who occasionally writes scripts (probably poorly) that take too much of my shared host's CPU cycles, and which therefore crap out with a 500 error (ie, "you're hogging resources"--I've had them confirm that).

For example, I wrote a script to create "zoomify" tiles on the fly...this can entail creating 100 cropped images (or more), a real processing hog. If I run it early in the AM when I'm the only guy on the box, it does fine. If there is any other load on the box, the script is killed.

What's the best strategy to avoid these sort of errors? Am I wrong in assuming that trying a PHP-based solution isn't going to work--PHP timeouts are measured over the scope of the single session, so a PHP-based controller will trigger a timeout (30 seconds on my host)? Is the solution client-side scripting firing smaller chunks of PHP code? Frames talking to each other? Something obvious I'm clueless about?

When you've stopped laughing, any pointers or resources would be appreciated.
posted by maxwelton to Computers & Internet (13 answers total)
 
Response by poster: I meant to add that the example script is fairly well streamlined, in that it destroys all of the images as its done with them, etc. This isn't the only script where this has happened to me--another example would be writing a large batch of simple text files which cannot be accomplished before PHP times out.

(I feel like I'm sitting in a car but haven't a clue what I need to turn, push or pull to get it running.)
posted by maxwelton at 9:08 PM on June 18, 2007


Can you do the processing on a local machine and then upload the results?
posted by null terminated at 9:23 PM on June 18, 2007


I think the answer is that you need to change tools, concepts, and goals.

For instance, the zoomify thing you're talking about would be better handled client-side using either JavaScript or even better, Flash.

And this: "Doctor, it hurts when I do that." "So don't do that." If you're having trouble making certain things work, maybe you need to take a step back and decide whether those are things you really need to do. Perhaps there's a different, less abusive thing you could do which would look different, feel different, but accomplish the same deep goal.
posted by Steven C. Den Beste at 9:26 PM on June 18, 2007


When I need to do something that I can't do reliably all in one script run (e.g. email 2000 users), I make a script that will do a portion of it (e.g. email 10 users) and remember where it left off (e.g. give each user a last_emailed timestamp). Then I set up a cron job to run the script periodically (e.g. every five minutes).
posted by scottreynen at 9:36 PM on June 18, 2007


Similar concept maybe... I developed a newsletter delivery thing on a shared server for a client. They have thousand of subscribers, and delivery is handled by a single PHP script. I encountered the same time-out problems as the processes were taking too long. Some suggestions that may help you:

1. Do the PHP set_time_limit thing.

2. Similar to the "talking frames" idea - you can break up your process into batches. My solution was to process a small batch at a time, then it will call the same script again, but this time sending in a variables to tell it where to start processing from. And so on.

3. Extending on the idea above, you can break up your jobs using CRON. Set it to run at intervals of like 5 minutes, and each time, process a small batch. For best results, schedule it to run only during off peak hours, so you reduce the chances of the admins suspending your account in case you're hogging up too much resources.
posted by arrowhead at 9:39 PM on June 18, 2007


Response by poster: scottyreynen and arrowhead, I've done very similar things as well, so maybe I was on to the right approach...I just wanted to be sure I'm not missing something obvious.

null, that's what we're doing now. But the images are loaded into a CMS, and in an ideal world the client checks a box that says "zoomify this image" and then it happens automagically.

Steven, thanks for the ideas. Javascript might be the answer, and maybe a flash utility might do the trick. I really like presenting HTML only to my CMS clients, but maybe I need to think outside the box a bit.
posted by maxwelton at 10:37 PM on June 18, 2007


Max, you should probably switch to a dedicated host. It's difficult to get around intentional resource roadblocks on a well-run shared host, and it sounds like your needs have grown to that point if you're writing server-side image-processing scripts anyway.

Arrowhead's cron idea will work nicely if your users don't mind having to wait a potentially long time.
posted by migurski at 11:16 PM on June 18, 2007


A quick follow up: a 500 error makes it sound like your image-chopping script is being run within the webserver process. You can use this to your advantage.

If you have a way of storing state (for example, session + database), have your script perform small amounts of the job on each request as Arrowhead suggests. Instead of letting it hit the script time limit, stop after 5-10 seconds and return a small chunk of progress information, e.g. "50 of 900 images completed". If you stick this in a frame and cause it to reload itself when it finishes (javascript), your CMS users will see a useful progress report as the image is chopped into pieces.
posted by migurski at 11:26 PM on June 18, 2007


Why are you destroying all the images as you're done with them? Are they really so unlikely to be used again, ever, that you couldn't implement a server-side image cache, and simply serve up pre-zoomed images instead of having to repeat all the zoom processing every time?

If you've got the storage space to do this, you might consider never serving up anything except already-cached zoomed images, and use a cron job or some other low-priority background thingummibob to generate the cache when there's not much else going on.
posted by flabdablet at 1:44 AM on June 19, 2007


I'm seconding flabdablet's suggestion. Hard drive space is often much cheaper than processor time, and the caching will deliver a faster, more consistent user experience.

It's even conceivable that you could do the image processing on a local machine. Have it connect via ftp, look for files that haven't been processed, download them, process them, and upload the results.

You should definitely do a quick estimation of how much space it would take to cache everything.
posted by Rictic at 5:01 AM on June 19, 2007


fladlabet/rictic -- I think what was meant was that the script is destroying the copies of the images in-memory so as to not be a memory hog on the server. i hope the tiles are cached.
posted by misterbrandt at 9:33 AM on June 19, 2007


Response by poster: Yeah, obviously the tiles are saved to the server, the script only needs to run once per image.
posted by maxwelton at 1:23 PM on June 19, 2007


In that case, just have your image-submission script add an entry to a queue, and use a lower-priority background process or a cron job to turn queue entries into zoomed tiles whenever there's spare CPU power available.
posted by flabdablet at 6:36 PM on June 19, 2007


« Older dSLR or not dSLR, That is the Question.   |   How to find a cheap doctor Newer »
This thread is closed to new comments.