Comments on: WinXP Scripting

Question: WinXP Scripting

bingo — Sun, 08 Aug 2004 07:15:58 -0800

What's a relatively easy way to script browser actions (events?) in Windows XP? (mi)

All I want to do is automate the act of going to a website, logging in, and clicking on certain links, resulting in the download of a file that must be saved with a name that I designate in advance. And then, do it again about ten more times, choosing different files to download each time, and of course giving them different names each time. The web interface will always be exactly the same, so will the choices.

If at all possible, the browser would be the latest IE or Netscape 7.2.

Note that I am not looking for some kind of slurping tool that will download the whole site, and everything linked to it.

By: bingo

bingo — Sun, 08 Aug 2004 07:29:45 -0800

For clarity: the "certain links" are always exactly the same. The layout of the website is always exactly the same. Occasionally, there may be one more choice to click on in a list of downloads, but not very often, and that's all the variety there is.

The downloaded files are .csv files which will, in a perfect world, be handed off to an excel macro that will bring them to their actual purpose.

By: cheaily

cheaily — Sun, 08 Aug 2004 08:36:04 -0800

I'm assuming that the CSV files are generated by the website?

I don't think it's possible to script that much detail, even using the windows scripting host in XP.

Depending on how the website is set up (or, even if you have access to modify the site yourself), you might be able to fudge the URL to automatically pick the options you require for each file.

If you want, email me, or post some more details.

By: jragon

jragon — Sun, 08 Aug 2004 08:40:20 -0800

Other than the incredibly expensive WinRunner, I can only think of Mac and Unix solutions. If you install curl, it can do a lot of this logic, especially if the site is barely changing.

By: five fresh fish

five fresh fish — Sun, 08 Aug 2004 09:29:34 -0800

I did it with Python, Bingo. I'm looking to move, so I wanted to scrape the MLS.CA website for new listings every day. Turns out to be dead easy! I'm sure the code is inefficient and ugly, but it took less than two hours for me to go from zero to full-speed. I love Python!

from BeautifulSoup import BeautifulSoup

import urllib2, ClientCookie, pickle, webbrowser



def MakeSoup(pageno):

    #fetch a page and make it into delicious soup

    urlstart = 'XXXobscuredXXX'

    urlend = 'XXXobscuredXXX'



    request = urllib2.Request(urlstart+str(pageno)+urlend)



    request.add_header('Accept-charset','utf-8,*')

    request.add_header('Cookie',"LegalDisclaimer=1")



    f = ClientCookie.urlopen(request)

    response = f.read()

    f.close()



    soup.feed(response)

    return soup





def ScrapeNumbers():

    #scrape the numbers out of the soup

    numlist = []

    idlist = soup.fetch('div', {'class': 'Label'})



    for i in idlist:

        s = str(i.contents[0])

        start = s.find('MLS')

        end = s.find('')



        if start > 0:

            #got an mlsno

            mlsno = s[start+10:end].strip()

            #get a property id, too

            start = s.find('PropertyID')

            end = s.find('">MLS')

            propid = s[start+11:end].strip()

            numlist.append((mlsno,propid))

    return numlist





#########################

# load our MLS number history

try:

    mlshistory = pickle.load(open('mlslist.pickle'))

except:

    mlshistory = []



# first pass gets the page count

print "Getting page count..."

pageno = 1

soup = BeautifulSoup()

soup = MakeSoup(pageno)

# identify page count

pagelist = soup.first('span', {'class': 'PageHeader'})

s = str(pagelist.contents[0])

start = s.find('of')

end = s.find('-')

pagecount = s[start+2:end].strip()

print "There are "+str(pagecount)+" pages"



# make more soup

while int(pageno) < int(pagecount):br>
    print "Processing page "+str(pageno)

    pageno += 1

    soup = MakeSoup(pageno)



# parse out new numbers

newnumbers = []

numlist = ScrapeNumbers()

for i in numlist:

    if i not in mlshistory:

        newnumbers.append(i)



print "New Numbers:"

print newnumbers



for i in newnumbers:

    mlshistory.append(i)

    webbrowser.open('XXXobscuredXXX'+str(i[1]),1)



#save the updated MLS number history

pickle.dump(mlshistory,open('mlslist.pickle','w'))

Dunno why it's double-spacing. Makes no real difference.

By: bingo

bingo — Sun, 08 Aug 2004 10:06:47 -0800

fff: Thanks, but I don't know enough about coding to read that (uncommented) well enough to use it in my situation.

cheaily: Playing with the URL is an interesting idea.

The csv files are generated by the website, but I don't need the script to actually do anything with them other than download them. If the handing off to the excel macro has to be done manually and that's the worst of my problems, then I'll be fine.

Here's what happens when I do it manually: I go to the site, which I have designated as a shortcut in IE. The username/password box comes up immediately, and IE fills it in for me. I click ok, and am presented with a page full of links that offer me choices for what kind of file I want to create. I click on blah, I click on blah, and wallah, the information is displayed before me. To show that I want to download it as a csv file, I click on blah. Windows asks me if I want to open the file, or save it. I say that I want to save it. Windows asks me what name to save it under. I tell it. The deed is done. I then go back to the website and choose some more options, and do it again.

Surely, to a browser, these links all have numbers, or some other labels, that can be remembered and used to find the same links every time? I imagine (in my non-programmer mind) code that does something like this (This is me talking to the browser):

a) go to the main page
b) enter username and password, click ok
c) You will see 20 links. Click on link #14.
d) You will see 5 links. Click on link #2.
e) You will see 7 links and three buttons. Click on button #1.
f) You will get a choice of whether to open the file, or save it. Choose save.
g) You will be asked what to name it and where to put it. Call it "bingo's file #8" and put it in the folder called "bingo's automatically downloaded csv documents."
h) Return to step c), but this time start with link #15 instead of #14.
i) Repeat until you've gone through the cycle starting with links 14, 15, 16, 17, and 18.

Then, in a perfect world, the excel macro will spring into action without a human having to be there to start it. But just steps a through i would make my life a lot easier.

Thanks.

By: nicwolff

nicwolff — Sun, 08 Aug 2004 14:40:32 -0800

Perl, HTTP::Recorder, and WWW::Mechanize will do this quite easily and nicely.

By: nicwolff

nicwolff — Sun, 08 Aug 2004 14:42:55 -0800

Or since you're on Windows I should have linked Perl.

By: holloway

holloway — Sun, 08 Aug 2004 15:37:03 -0800

It's not using IE but Canoo takes an XML file of events (go here, click that, download) and plays that back.

By: bingo

bingo — Sun, 08 Aug 2004 16:24:34 -0800

As far as Canoo, the website won't cooperate with anything that doesn't identify itself as a recent version of IE.

nicwolff: Thanks, I guess you've shown me that for someone at my level (programming novice; all I know is some VB and bash shell stuff), this is going to take a lot of reading on my part, even if the result would be simple to acheive for someone who knows enough.

But surely then, this is a gap in the market waiting to be filled? Surely there are others like me, wanting to automate their browsing experience at a fairly simple level, but not knowing python or perl (and, in general, not needing them for my job)?

By: holloway

holloway — Sun, 08 Aug 2004 20:51:42 -0800

Say what? Canoo's website works for me in Firefox 0.92 / WinXP -- and as you can see from Google, canoo is very popular.

By: bingo

bingo — Mon, 09 Aug 2004 17:23:33 -0800

holloway, I'm not talking about the website, I'm talking about the product. I have to use either IE or Netscape for the task.