Ask MetaFilter questions tagged with cron

How to automate backups on my ftp server?

stupidsexyFlanders — Sat, 05 Dec 2009 12:29:34 -0800

I have some wordpress sites hosted at bluehost that I want to keep backed up. Can you help me automate this? Detailed needs inside. What I've been doing is going to the bluehost control panel every few weeks and manually downloading, first, all the files on the ftp server. Then I'm separately downloading all the dbs. Then I'm throwing them all in a folder on my local machine and have Carbonite back them up remotely. I keep five backup sets, and delete the oldest when I add a new one.

I would like to automate this 100% if possible -- maybe have some kind of script that every week zips up the backups into a one big file on the ftp server, which I can then use SynchBack Pro to mirror to my local HD on a regular basis?

The solution would have to compress all the dbs and a defined portion of the ftp file structure into one file named 120509.* (itself inside a folder called /backups). The following week, same exact thing, except the files are compressed into 121209.*, etc. Each week it should look in /backups for a file named with a date older than 5 weeks and delete it.

Then I would use Synchback every week to pull down the contents of /backups on the ftp server to my local drive, which I will tell Carbonite to pull up remotely.

What do I do to automate the compressing and saving and sequential naming and date-based deleting on the ftp server?

I've never used cron or SSH and only done super basic stuff using phpmysql. I've read through these two askmes and while they did include code snippets that seemed like what I needed, I don't know what to do with them or how to modify them to do what I specifically need. So speak very slowly. Thanks!

PHP -> Twitter API at specified times: what am I Doing Wrong?

goodnewsfortheinsane — Wed, 29 Jul 2009 18:29:11 -0800

Please help me write a PHP script that publishes Twitter updates at pre-specified times. I'm working on a script that will update Twitter with reminders about upcoming shows at a music festival, building on lodev's idea here.

I've currently got this script, which calls a text file structured like in this made-up example:

XXXUPDATE SponsorCo Stage: THE BEATLES in 30 minutes XXXTIME August 21 2009 21:30 +0200 XXXDONE 0 XXXENDLINE XXXUPDATE MegaCorp Stage: ROLLING STONES in 30 minutes XXXTIME August 21 2009 22:45 +0200 XXXDONE 0 XXXENDLINE

Proprietary format, I know. :) There are six Twitter accounts right now for separate concert stages, more may follow.

All six separate scripts are called from a Ruby file via pycron, every minute. I understand that's plenty often and I don't really need that kind of temporal resolution, but it uses relatively few processor cycles so I'm not really complaining.

As you can see in the PHP code:

if (($now <= ($utime + 30)) && ($now >= ($utime - 29))) {

if ($update[3] !== 1) {

When an update is ready for publication it should, theoretically at least, satisfy the first "if" statement and be published. The second "if" checks a "done" flag in the text file which I have as of yet not used (i.e. they're all set to 0 and the current code doesn't amend them).

Now, I "stress-tested" this with some test updates on all six accounts, including many simultaneous ones (carrying the same timestamp) on the different accounts for ten minutes with six (one for each account) every minute, and out of maybe a hundred-something tweets (I ran multiple iterations of the test) only one was dropped, i.e. it was in the text file but was not published.

Even one dropped update is unacceptable to me, and I strongly suspect I am Doing It Wrong. But how? The problem basically boils down to the following question:

What better method would there be to push out the updates reliably using the server config I currently have (XP, Apache, PHP5, Ruby, pycron), very preferably without using databases?

When it comes to scripting I really only speak PHP and a little JavaScript: even the (very basic) Ruby script was kindly provided to me for a different project.

So there's just too little time before the event to learn a new language or start out with MySQL or whatever before the event, and I'm not really eager to change to different software or change my server configuration (or set up another box) just for one-off project that will last just a few days. This is just to say that it's not due to an unwillingness on my part, but rather due to practical considerations.

If anyone could offer their insights I would be ever so grateful. Thanks in advance, guys.

Cronjobs

uncle harold — Wed, 09 Jul 2008 00:48:01 -0800

Cron Jobs - separate processes? This seems a pretty simple question, but I somehow can't find any info on this: Is every job launched by a (Linux) cron daemon started as a new process? What happens in the (theoretical) case where a repeating tasks overlaps with its own next iteration?

My website's scripts eat up too much bandwidth. How do I resolve this?

goodnewsfortheinsane — Sat, 05 Apr 2008 09:21:43 -0800

My site's cron tasks eat up a lot of bandwidth. Can I change the paths from http to local? Or can I move the tasks to my home server? One of my websites (the election one) runs a truckload of PHP to parse, mesh, filter, and cache an equally large number of external RSS feeds which are then stored and included in a reasonably lean (~10k) index.html. This creates a strange situation where the bandwidth consumed by the PHP (locally, but via http) far exceeds the actual bandwidth consumed by visitors. My web hosting provider has warned me about the excessive activity, but given the limited number of visitors, upgrading to a hosting deal with a higher bandwidth limit isn't really worthwhile for me.

How do I resolve this?

Current situation: a bunch of FeedForAll PHP scripts are executed using a Ruby-as-CGI script (which wackybrit kindly helped me with) called via cron at regular intervals, varying by nature of content: none-time critical content would be updated hourly, whereas latest news etc. is updated every 10 minutes (was 2 minutes, but I took it down a notch for now to avoid further problems with my host). My monthly bandwidth limit is 8 GB; actual visitor usage is a modest fraction of that, but the scripting alone was responsible for 37 GB (!) in March.

Several options:

1) The bandwidth is measured by the host using HTTP responses. If I run the scripts using local UNIX paths, e.g. /usr/gnfti/www/ etc., the scripts generate no HTTP headers and thus don't count towards the quota. Of course they will still incur server load, but I have discussed this with the hosting company and it seems they're okay with it if I limit the use somewhat as a compromise. But then, not all of these scripts seem to be okay with local paths, most notably the keyword filter (rssFilter.php), which is responsible for 25% of the bandwidth on its own.

Any ideas how to make it work this way anyway?

2) I have a home server running WAMP on XP. Could I run the PHP on this box instead and have the output automatically upload to the remote server via FTP? Most of the bandwidth is to do with moving stuff around locally anyway; the actual output is in the order of kilobytes. I imagine this would involve running some sort of cron + FTP client/scheduler on the home box, but my knowledge in this area is limited.

Could this be done, and if so, how?

3) Lastly, if you have any suggestions on how to pull this off aside from the examples above, that too would be much appreciated.

Thanks in advance for any suggestions or insight you might have to offer, guys.

Help with cron

tylermoody — Fri, 03 Aug 2007 19:41:52 -0800

How do I make a command-line script run at startup/login in OSX? I want to make rkhunter run while I'm asleep. I've found a script to get it to update, run, and write to a temp logfile, but this howto just tells me to run it as a cron job. I'm trying to figure out cron, but that tutorial tells me to run sudo cron as a startup script. That's what I want to use cron for!

What do I have to do to get the cron daemon going every time I log in? Both short answers and links to long, way-too-much-information texts are welcome.

Questions about cron

zek — Wed, 21 Feb 2007 14:57:54 -0800

Paging Linux geeks -- please teach me the inner mysteries of cron and scripting. Simple questions follow. Right now I have a Windows machine that downloads information off the web every hour. This computer is a 10-year old notebook computer that's about to fall apart, and I have a new computer on the way to replace it. I badly need a Linux system to run some institutional science apps, so it looks like I should go the Linux route.

Currently, my Windows scripting is orchestrated by a simple hand-coded application whose sole job is to call a different Windows batch file at 10 minutes past every hour (GET00.BAT, GET01.BAT .... GET23.BAT). Each batch file calls wget to download whatever data is needed. It works splendidly.

However it's not enough just to call the batch file... my app also has to form the pieces of the date/time and call them as arguments. For example today 2/21/07 4:00pm would be called as:
GET00.BAT 2007 02 21 16 00
(note that some are zero-padded). These date/time values are needed to construct target filenames and are used in some source URLs. For example, the command WGET blah.com/junk.dat -O %1-%2-%3 would put my data into 2007-02-21.dat.

So my questions are essentially:

1) Is cron appropriate for this kind of work?
2) How can I set up a cron job where it runs all the time (i.e. when the computer is always on)? I'm not clear on how daemons are set up in a Linux desktop environment.
3) Can scripting devise these required time parameters explained above, or can cron pass them? Or is an ugly C++ program needed?
4) A real newbie question: Does my choice of a shell (ksh, csh, etc) matter much here?

Fortunately the data is not critical, so it's no biggie if I make some mistakes getting this set up. I'm just hoping for a push in the right direction. Thanks.

How can I preserve Unicode characters in `mysqdump`?

cmyers — Mon, 28 Aug 2006 11:55:32 -0800

How can I make backups of a MySQL database (via mysqldump) and preserve Unicode characters (such as em dashes)? I use mysqldump and rsync to make backups of my TextDrive account. One of the forums I run, however, often includes unencoded Unicode entities, such as em dashes and bullets.

Because these characters aren't straight ASCII, restoring from previous mysqldump backups has mangled them, producing gibberish but preserving the other, straight ASCII text in the database.

My current mysqldump call looks like this:

/usr/local/bin/mysqldump -uuser -ppassword --quote-names --complete-insert --extended-insert --quick --lock-tables=false --skip-add-locks --all-databases | gzip > db.sql

. What's necessary to preserve these Unicode characters?

Thanks.