Comments on: How can I automate this web browsing task?

Question: How can I automate this web browsing task?

mebibyte — Fri, 29 Aug 2008 12:49:22 -0800

Please help me automate a repetitive task (log in to website, then after successful login, load some pages, then log out), in a way that can be scheduled so that it runs at two different hours each day.

Every day, I need to log on to a website. This part is important, as without logging in, I can't do the actions that I want to do.

After logging in, I need to load several specific pages. The pages that need to be loaded can be determined in one of two ways:

1. By programmatically loading one page, retrieving all links that say 'Add', and loading the pages referenced by those links
2. I can manually add the links to a list (or a Firefox bookmark, which is how I've been semi-automating this)

Once that's done, the script (or whatever) can log out.

I'm good with computers. I program for a living and am familiar with a bunch of scripting languages. The reason I'm asking for this is, where I currently work, I am expected to load a page twice a day (*exactly* when a catalogue refreshes, it's time-sensitive) and Add a list of products (it uses HTTP GET so I just need to follow links). It really only takes a couple of minutes, but I hate switching from one task to another and back.

I don't mind manually finding the links on my own time. I just don't want to be obligated to be online at EXACTLY the time that their catalogue refreshes.

Here's what I've come up with so far:

1. (ideal) manually adding the links to a firefox bookmark, then using some sort of firefox extension/applescript or something to use my saved login/password to log in, then load the bookmarks at a time I specify

I would like to stress that (1) would be ideal.

2. Use www:mechanize to retrieve the Add links, and then follow them using threaded requests.

Any other suggestions, or any tools built specifically for this task?

Thanks!

By: blind.wombat

blind.wombat — Fri, 29 Aug 2008 12:59:05 -0800

You might be able to do this with autoit if you can't find a way to do it all with browser extensions/addons.

By: mebibyte

mebibyte — Fri, 29 Aug 2008 13:10:43 -0800

Sorry! That reminds me. I work with Mac and Linux computers only. No VMs, no wine.

By: knowles

knowles — Fri, 29 Aug 2008 13:12:06 -0800

Can't you do this with Applescript, Automator, or a cronjob?

By: XMLicious

XMLicious — Fri, 29 Aug 2008 13:13:19 -0800

PushToTest is a Java-based (runs on Linux and Windows, I don't know about OSX) web application testing framework that can be used to script things like this. It's designed so that the scripts can be packaged up and run automatically by a Unix daemon, I believe.

(But actually, you ought to be able to write something to do this from scratch in any language you're familiar with, if you feel up to the challenge.)

By: pdxpatzer

pdxpatzer — Fri, 29 Aug 2008 13:24:11 -0800

you can use any of the following ...

cURL
http://curl.haxx.se/

Watir
http://wtr.rubyforge.org/

Selenium
http://selenium.openqa.org/

same examples of web client programming can be found here (the content is old but it should still be accurate)
http://oreilly.com/openbook/webclient/

hope this helps

By: adamrice

adamrice — Fri, 29 Aug 2008 13:25:24 -0800

iMacros for Firefox
Automating Firefox

By: XMLicious

XMLicious — Fri, 29 Aug 2008 13:27:58 -0800

Whoa! Selenium looks sweet. Thanks for posting that, pdxpatzer.

By: austinetsu

austinetsu — Fri, 29 Aug 2008 13:28:30 -0800

I'll 2nd Watir. It was created to do automated testing of web apps with Internet Explorer. However we've been using it at work to script automated tasks on the web

By: hooray

hooray — Fri, 29 Aug 2008 13:38:16 -0800

You can do this with Applescript and Safari. There's a "do javascript" command in the Safari library, that will help you with a lot of this stuff --- filling in forms, submitting, grabbing links. I find macosxhints to be a useful resource for this kind of thing.

By: rjt

rjt — Fri, 29 Aug 2008 13:47:31 -0800

Untested, but something like this should work...



cd $HOME/tmp/



wget --save-cookies cookies.txt --post-data 'username=foo&password=bar' http://www.example.com/login



wget --load-cookies cookies.txt --recursive --max-depth 0 --delete-after http://www.example.com/your/page

Note that the second wget will follow all links on the page, not just the ones you want. This may or may not be a problem. (Also the --domains switch might help here.)

By: rjt

rjt — Fri, 29 Aug 2008 13:57:02 -0800

sorry, that should be --max-depth 1

By: wongcorgi

wongcorgi — Fri, 29 Aug 2008 14:42:29 -0800

Seconding a curl or wget script on a cron job

By: PueExMachina

PueExMachina — Fri, 29 Aug 2008 17:08:48 -0800

Ruby's WWW:Mechanize is well documented and easy to use. I don't see why you couldn't write a short script to do what you want and run it from cron.

By: Happy Dave

Happy Dave — Fri, 29 Aug 2008 17:21:15 -0800

I've done something similar to this using a combination of Aurora and Automator. Basically, each morning, Aurora wakes the mac, plays a song from an iTunes playlist, then opens safari, loads the page for NPR's The writer's Almanac (I'm in the UK, can't get it on the radio), finds the link on the page that has 'play' in it and clicks it.

This works because as well as hooking into iTunes, Aurora has a field that you can drag and drop any file or application to in order to launch it at a certain time. And you can save Automator scripts as Application files.

By: flabdablet

flabdablet — Sat, 30 Aug 2008 05:50:55 -0800

I'd use wget, bash and cron.