How can I run semi-complex automated processes (FTP, ZIP etc) on Windows PCs?
November 2, 2010 4:01 AM Subscribe
How can I run semi-complex automated processes (FTP, ZIP etc) on Windows PCs?
We have a bunch of computers on the other side of the world. When these computers are in use (which could be every day for a month, and then a break of two months) they produce text files in a directory.
What I want to do ideally is:
- When the text file is created in directory A, it is then FTPed to a remote server.
- If this is successful, the file is moved to directory B
- Every month, files in directory B are zipped up (possibly with the zip file also FTPed to the server) and the zip placed in directory C with the files moved to directory D.
There are probably a dozen different ways of doing this, but what I'm looking for is
a) reliability - it's a pain to access these computers, so something that doesn't require too much checking, especially if these computers are restarted is important. Bonus for simplicity in maintaining.
b) it works well on failure: e.g. if the FTP fails, it doesn't move the file to directory B; if a file with the same name exists in B it handles it gracefully etc. Bonus points if it could do something like send an email at the end of the day with any failure/error messages (so not bombard me with a 1000 emails if FTP is down for two days)
c) these computers could be running different versions of Windows and have other quirks, so any solution that has a minimal number of requirements would be great.
At the moment I'm looking at oldish version of VisualCron which I have a site licence for as a solution, but that now crashes on launch which isn't endearing itself to me.
We have a bunch of computers on the other side of the world. When these computers are in use (which could be every day for a month, and then a break of two months) they produce text files in a directory.
What I want to do ideally is:
- When the text file is created in directory A, it is then FTPed to a remote server.
- If this is successful, the file is moved to directory B
- Every month, files in directory B are zipped up (possibly with the zip file also FTPed to the server) and the zip placed in directory C with the files moved to directory D.
There are probably a dozen different ways of doing this, but what I'm looking for is
a) reliability - it's a pain to access these computers, so something that doesn't require too much checking, especially if these computers are restarted is important. Bonus for simplicity in maintaining.
b) it works well on failure: e.g. if the FTP fails, it doesn't move the file to directory B; if a file with the same name exists in B it handles it gracefully etc. Bonus points if it could do something like send an email at the end of the day with any failure/error messages (so not bombard me with a 1000 emails if FTP is down for two days)
c) these computers could be running different versions of Windows and have other quirks, so any solution that has a minimal number of requirements would be great.
At the moment I'm looking at oldish version of VisualCron which I have a site licence for as a solution, but that now crashes on launch which isn't endearing itself to me.
One method would be to install something like Strawberry Perl and write a perl script yourself that would do what you want. There are plenty of modules to handle things such as connecting and uploading files to a secure FTP server, handling issues that arise and zipping up files.
For example, here is the first part of the code (untested, written from memory) which will check a file exists and then attempt to upload it to an FTP server. Note that if anything fails, the script will die so the file will be re-attempted the next time it is run (which may be what you don't want):
use Net::FTP;
use strict;
my $file = "name_of_file.txt";
my $host = "some.host.com";
my $username = "theusername";
my $password = "thepassword";
my $dir = "/path/to/the/directory/";
# Exit unless the file exists
exit unless (-e $file);
# FTP the file
my $ftp = Net::FTP->new($host) || die "Can't connect to $host: $!";
$ftp->login($username, $password) || die "Can't log in: $!";
$ftp->cwd($dir) || die "Can't cwd to $dir: $!";
$ftp->put($file) || die "Upload of $file failed: $!";
$ftp->quit;
# If we are here, everything was fine so do the moving/zipping/etc.
Downsides are that you'll need to learn Perl and have a fairly hefty installation for what would essentially be about a 5k solution.
It may be possible in DOS and with some specialised command line applications (for example, IzArc has a command line version for zipping up files and there is probably one for FTP'ing too) although you'd still have to learn how to write the batch files.
posted by mr_silver at 4:33 AM on November 2, 2010
For example, here is the first part of the code (untested, written from memory) which will check a file exists and then attempt to upload it to an FTP server. Note that if anything fails, the script will die so the file will be re-attempted the next time it is run (which may be what you don't want):
use Net::FTP;
use strict;
my $file = "name_of_file.txt";
my $host = "some.host.com";
my $username = "theusername";
my $password = "thepassword";
my $dir = "/path/to/the/directory/";
# Exit unless the file exists
exit unless (-e $file);
# FTP the file
my $ftp = Net::FTP->new($host) || die "Can't connect to $host: $!";
$ftp->login($username, $password) || die "Can't log in: $!";
$ftp->cwd($dir) || die "Can't cwd to $dir: $!";
$ftp->put($file) || die "Upload of $file failed: $!";
$ftp->quit;
# If we are here, everything was fine so do the moving/zipping/etc.
Downsides are that you'll need to learn Perl and have a fairly hefty installation for what would essentially be about a 5k solution.
It may be possible in DOS and with some specialised command line applications (for example, IzArc has a command line version for zipping up files and there is probably one for FTP'ing too) although you'd still have to learn how to write the batch files.
posted by mr_silver at 4:33 AM on November 2, 2010
All of this stuff is possible using SyncBackSE, i.e. you could create a profile (a backup script) to:
1) Back up all files from A to a server via FTP
then
2) Run a batch file on success, moving all the files to directory B
Then you'd have a separate profile to
1) Back up files from dir B to a zip in dir C
then
2) Run another batch file run on success to move the files from B to D.
Batch files to copy from one folder to another in Windows are pretty straightforward (you can use the xcopy command).
The key thing is that the profiles in SyncBackSE can be configured to only run the batch files on successful completion of the FTP/ZIP backups. You can also have it email you the log file after each profile runs.
posted by le morte de bea arthur at 4:44 AM on November 2, 2010
1) Back up all files from A to a server via FTP
then
2) Run a batch file on success, moving all the files to directory B
Then you'd have a separate profile to
1) Back up files from dir B to a zip in dir C
then
2) Run another batch file run on success to move the files from B to D.
Batch files to copy from one folder to another in Windows are pretty straightforward (you can use the xcopy command).
The key thing is that the profiles in SyncBackSE can be configured to only run the batch files on successful completion of the FTP/ZIP backups. You can also have it email you the log file after each profile runs.
posted by le morte de bea arthur at 4:44 AM on November 2, 2010
Googling shows that rsync, the premier unix tool for copying and syncing files over a network, is available in several versions and ports for Windows and there are lots of different pages giving instructions for using it with FTP.
posted by XMLicious at 4:59 AM on November 2, 2010 [1 favorite]
posted by XMLicious at 4:59 AM on November 2, 2010 [1 favorite]
Umm... I don't know if this is kosher or not... but the company I work for makes a free product called Mojo which is a workflow tool - a tool that executes a bunch of activities and analyzes the outcome of those activities to figure out whether it should execute the next activity.
It has scheduling, email activities, and continuous integration (i.e. continuously checking a directory to see if there have been any changes, and then executing a set of activities if there has been a change).
The product might be overkill for what you're looking to do, and has a high learning curve (but a lot less difficult than learning Perl/Python) but it's free, and does what you want.
posted by baxter_ilion at 6:09 AM on November 2, 2010
It has scheduling, email activities, and continuous integration (i.e. continuously checking a directory to see if there have been any changes, and then executing a set of activities if there has been a change).
The product might be overkill for what you're looking to do, and has a high learning curve (but a lot less difficult than learning Perl/Python) but it's free, and does what you want.
posted by baxter_ilion at 6:09 AM on November 2, 2010
You have a few options but they all require some level of scripting knowledge. You could use Autohotkey and the built in command line ftp client or a windows port of rsync. Somewhere in there you'll need to put in functionality to verify the copy was made. I typically use command line SMTP to send me a notice when an exception occurs.
posted by damn dirty ape at 7:12 AM on November 2, 2010
posted by damn dirty ape at 7:12 AM on November 2, 2010
Have you considered using batch files (.bat) and Windows Task Scheduler? It is pretty old tech, but I've found it more reliable than some other, fancier options.
Here is an example of a batch script that will upload a file to an FTP server:
posted by Gainesvillain at 8:31 AM on November 2, 2010
Here is an example of a batch script that will upload a file to an FTP server:
echo user [USERNAME]> ftpcmd.dat echo [PASSWORD]>> ftpcmd.dat echo bin>> ftpcmd.dat echo cd [DIR WHERE FILE IS LOCALLY]>>ftpcmd.dat echo put [LOCAL FILE]>> ftpcmd.dat echo quit>> ftpcmd.dat ftp -n -s:ftpcmd.dat [FTP SERVER ADDRESS] del ftpcmd.dat
posted by Gainesvillain at 8:31 AM on November 2, 2010
Forgot about the .zip file. The Windows Task Scheduler can run scripts at long intervals. Creating a .zip file from the contents of a directory via a batch script should be pretty easy. So should moving that file. You can even do file names with timestamps. Then you just have the scheduler run that script once a month.
Timestamped file name example, fo sho untested:
posted by Gainesvillain at 8:57 AM on November 2, 2010
Timestamped file name example, fo sho untested:
for /f "tokens=1 delims=: " %%h in ('time /T') do set hour=%%h
for /f "tokens=2 delims=: " %%m in ('time /T') do set minutes=%%m
for /f "tokens=3 delims=: " %%a in ('time /T') do set ampm=%%a
cd [LOCAL DIRECTORY]
compact *.* [FILENAME]_%date:~10,4%%date:~4,2%%date:~7,2%_%ampm%%hour%%minutes%.zip
xcopy [FILENAME]_%date:~10,4%%date:~4,2%%date:~7,2%_%ampm%%hour%%minutes%.zip [DESTINATION DIR]
del [FILENAME]_%date:~10,4%%date:~4,2%%date:~7,2%_%ampm%%hour%%minutes%.zip
posted by Gainesvillain at 8:57 AM on November 2, 2010
I don't understand what directory D is doing, but it appears you can do all of this with 'ftp.exe -s' and ERRORLEVEL in a batchfile, echo "foo success" >> logfile.txt and similar.
posted by rhizome at 10:44 AM on November 2, 2010
posted by rhizome at 10:44 AM on November 2, 2010
+1 to learning batch well enough to do this... should be fairly simple in your situation and may prove itself handy for other tasks down the road.
posted by RolandOfEld at 12:16 PM on November 2, 2010
posted by RolandOfEld at 12:16 PM on November 2, 2010
Response by poster: Thanks for the help; I've decided to try and go down the batch file route: if that doesn't work out, I'll have a look at SyncBackSE which does seem pretty well suited to the task.
posted by Hartster at 4:24 AM on November 4, 2010
posted by Hartster at 4:24 AM on November 4, 2010
This thread is closed to new comments.
posted by Luddite at 4:30 AM on November 2, 2010 [1 favorite]