My website's scripts eat up too much bandwidth. How do I resolve this?

goodnewsfortheinsane — Sat, 05 Apr 2008 09:21:43 -0800

My site's cron tasks eat up a lot of bandwidth. Can I change the paths from http to local? Or can I move the tasks to my home server? One of my websites (the election one) runs a truckload of PHP to parse, mesh, filter, and cache an equally large number of external RSS feeds which are then stored and included in a reasonably lean (~10k) index.html. This creates a strange situation where the bandwidth consumed by the PHP (locally, but via http) far exceeds the actual bandwidth consumed by visitors. My web hosting provider has warned me about the excessive activity, but given the limited number of visitors, upgrading to a hosting deal with a higher bandwidth limit isn't really worthwhile for me.

How do I resolve this?

Current situation: a bunch of FeedForAll PHP scripts are executed using a Ruby-as-CGI script (which wackybrit kindly helped me with) called via cron at regular intervals, varying by nature of content: none-time critical content would be updated hourly, whereas latest news etc. is updated every 10 minutes (was 2 minutes, but I took it down a notch for now to avoid further problems with my host). My monthly bandwidth limit is 8 GB; actual visitor usage is a modest fraction of that, but the scripting alone was responsible for 37 GB (!) in March.

Several options:

1) The bandwidth is measured by the host using HTTP responses. If I run the scripts using local UNIX paths, e.g. /usr/gnfti/www/ etc., the scripts generate no HTTP headers and thus don't count towards the quota. Of course they will still incur server load, but I have discussed this with the hosting company and it seems they're okay with it if I limit the use somewhat as a compromise. But then, not all of these scripts seem to be okay with local paths, most notably the keyword filter (rssFilter.php), which is responsible for 25% of the bandwidth on its own.

Any ideas how to make it work this way anyway?

2) I have a home server running WAMP on XP. Could I run the PHP on this box instead and have the output automatically upload to the remote server via FTP? Most of the bandwidth is to do with moving stuff around locally anyway; the actual output is in the order of kilobytes. I imagine this would involve running some sort of cron + FTP client/scheduler on the home box, but my knowledge in this area is limited.

Could this be done, and if so, how?

3) Lastly, if you have any suggestions on how to pull this off aside from the examples above, that too would be much appreciated.

Thanks in advance for any suggestions or insight you might have to offer, guys.

Ask MetaFilter questions tagged with rssfilter

My website's scripts eat up too much bandwidth. How do I resolve this?