(Very) regular retrieval of XML feed

August 12, 2006 3:18 AM

I have an XML feed which requires a username and password. I want to retrieve the contents every ten seconds or so (maybe thirty seconds at most) and store all the content that's retrieved. Is there a windows client out there that will do this? Or I just need to buckle down and code something....?
posted by humuhumu to Computers & Internet (10 answers total)

I think there's a windows cron, which should be able to do the scheduling for you, but I don't know of anything that would just fetch the file for you... telnet maybe.

Doing this on a *nix system would be a lot easier, if you have that option open to you.
posted by beerbajay at 4:03 AM on August 12, 2006

Windows: at and wget.
posted by orthogonality at 4:22 AM on August 12, 2006

Or, hell, just put it tin a sdcript and sleep until 30 secs have elapsed.
posted by orthogonality at 4:23 AM on August 12, 2006

I'm guessing the problem part of this isn't the "do something every thirty seconds" part, it's the "requres a password" part?

What kind of "requires a password"? Fill in a form on a web page and click a button? Or something like Apache authentication? Or something else?
posted by AmbroseChapel at 5:34 AM on August 12, 2006

You're likely to piss someone off if you don't get the server owner's permission before you start doing this. It'll be very obvious in the log file and very easy to block.
posted by cillit bang at 5:39 AM on August 12, 2006

Response by poster: No problems with the server owner - have his permission. User/pass is apache authentication: access the page and the usual dialogue box appears.
posted by humuhumu at 9:32 AM on August 12, 2006

http://username:[e-mail]/feed.xml
plus wget and windows scheduler?
posted by blag at 10:47 AM on August 12, 2006

I'm a very big fan of the winhttprequest component; you can script it very easily in vbscript, and it can handle many types of auth quite well. It's built into your windows client already. You could then put in the logic easily to give meaningful file names based on timestamp, etc. The basic script would be this, with any additional login info, headers, etc added in- you may need to look up the winhttprequest properties and methods on MSDN, but it's free and well documented.

' connect to server
Set myHttp = CreateObject("WinHttp.WinHttpRequest.5.1") ' on windows XP; if this errors out, try removing the .1
myHttp.Open "GET", "http://www.feedaddress.com/feed.xml", False
myHttp.Send
myContent = myHttp.ResponseText

' write to file
Set FSO = CreateObject("scripting.filesystemobject")
myFile = FSO.CreateTextFile("myfile.xml")
myfile.write myContent
myfile.close
Set FSO = nothing

You can then add rather quickly any looping, auth passing (as request headers) or timestamp-based filenames, checking for http status code 200 before writing the file, etc.

And, since windows scheduler doesn't expose sub-minute scheduling, you could set the vbscript to do its own looping, pausing 10 seconds after every retrieval, and just leave task scheduler to start it every hour- that way, if the script crashes and dies you're not losing more than one hour of potential downloaded files.
posted by hincandenza at 7:57 PM on August 12, 2006

I have to ask ... every ten seconds?
posted by AmbroseChapel at 12:24 AM on August 13, 2006

Response by poster: I have to ask ... every ten seconds?

Yup. It's data that changes very regularly, like the title says.

I think a wget and a cronjob is going to do it, with a bit of post-processing to dump the data into a file...
posted by humuhumu at 1:30 PM on August 13, 2006