A simple way to monitor a web page for changes?

January 13, 2009 12:55 PM

My employer has instituted a web browser based work request system. We’re suppose to monitor a web page and then immediately respond to work tickets when they appear. The problem is that days or even weeks can go by before a ticket appears, we have to use Internet Explorer, the page doesn’t refresh automatically, we have to keep logging back into the system every half-hour or so, and management doesn’t seem to have any interest in improving the system.

So, I’m looking for something that could monitor a web page (which will look the same except when a ticket is there), keep refreshing it automatically, and then sound some kind of an alarm (pop-up a window, sound an audible alarm, send an email or make a phone call) when a ticket appears. Oh, and we can’t spend any money.

I am not a professional computer geek, but if it helps, I do have some experience with writing simple computer programs and scripts.

posted by someonesomewhere to Computers & Internet (28 answers total)

(MaybeNotSo)Stupid Question- Doesn't your ticketing system generate an email when a new ticket is submitted? That's pretty basic functionality.
posted by mkultra at 1:02 PM on January 13, 2009

I can't help with much of your query but there is a free auto-refresh program for IE here.
posted by kate blank at 1:02 PM on January 13, 2009

Response by poster: No, the system does not generate an email and the person in charge of the system claims they can't do it.
posted by someonesomewhere at 1:04 PM on January 13, 2009

You could look at building script based around curl or wget to login, scrape the page, and alert based on the HTML output. since you're competent with scripting.
posted by rodgerd at 1:05 PM on January 13, 2009

Most application frameworks include HTTP request and response classes for getting data from the web and manipulating it. In .NET you have WebClient, WebRequest and WebResponse under System.Net. You can basically write a program to periodically retrieve the page and search the response string for some key HTML or text contained in every ticket, but is not contained in the page in its empty state.
posted by rocketpup at 1:07 PM on January 13, 2009 [1 favorite]

Too bad you have to use IE only, because this Firefox extension looks like just the ticket.

Firefox only (Windows/Mac/Linux): Check4Change is a simple Firefox extension that monitors web-pages for changes at a user-determined interval.

posted by notyou at 1:07 PM on January 13, 2009

There's an extension for Firefox that does exactly this: Update Scanner. Unfortunately for you, it only works for Firefox.
posted by arco at 1:10 PM on January 13, 2009

If you are comfortable with scripting, you might find the information here useful.
posted by timmow at 1:10 PM on January 13, 2009

This site claims to offer this service, for free. I've never used it, so this isn't necessarily an endorsement.
posted by arco at 1:11 PM on January 13, 2009

Well, why do you have to use IE? If it's just a matter of the site checking the user agent string, you can spoof that in Firefox and use the solutions described above.
posted by mr_roboto at 1:15 PM on January 13, 2009

I've worked with, and helped design, request systems like the one you are referring to. I have NEVER seen or worked with one that doesn't have the capability to email users based on a set of criteria (i.e.--email all new requests, email requests only from certain users, etc.).

That said, since I'm not familiar with your design, is your IT guy at least pursuing options? FWIW: It's not like you're asking for something completely unheard of.
posted by PsuDab93 at 1:15 PM on January 13, 2009

You could write something which retrieves the page every minute or so with curl or wget, and fires off an email when it changes — say, if the size changes, or the MD5 of its content changes, or it no longer contains the string "No tickets found", or something (whatever works and doesn't produce too many false positives). If you don't have a system that can run curl and a shell, you could write it in perl. There's almost certainly a perl script to do exactly what you're asking on CPAN already, but I don't know what it is.
posted by hattifattener at 1:18 PM on January 13, 2009

Low rent solution:

Build yourself a frameset or a page with a suitably large iframe in it, have another frame or the main window make a call to refresh the (i)frame with the page you're monitoring in every X minutes.
posted by mandal at 1:23 PM on January 13, 2009

Do you have to use Internet Explorer because (a) IT won't install anything else on your locked-down workstation or (b) this particular update page fails to work in non-IE browsers? Because if it's (a), you should still be able to run Portable Firefox from a USB stick or even from a folder inside your own user profile.
posted by flabdablet at 1:23 PM on January 13, 2009

Explain the problem to your managers in cost to the company. How much is the wasted time costing and what else could you be doing?
posted by Pants! at 1:34 PM on January 13, 2009

Ah, webmon.pl in WWW::Monitor.
posted by hattifattener at 1:34 PM on January 13, 2009

Even if it doesnt work properly with FF, you can still use FF to detect changes.
posted by damn dirty ape at 1:45 PM on January 13, 2009

I've worked with, and helped design, request systems like the one you are referring to. I have NEVER seen or worked with one that doesn't have the capability to email users based on a set of criteria (i.e.--email all new requests, email requests only from certain users, etc.).

That said, since I'm not familiar with your design, is your IT guy at least pursuing options? FWIW: It's not like you're asking for something completely unheard of.

I realize it won't help in the short term, but this bears repeating. It's like a car without seatbelts. I'm not even sure how they found a system that lacks this functionality.
posted by mkultra at 1:46 PM on January 13, 2009

Further to mkultra's last comment, what's the name of the system? It is not outside the bounds of possibility that your IT "experts" are wrong/lazy/both if they implemented such a nonsensical solution.
posted by Nick Verstayne at 2:06 PM on January 13, 2009

Ok, first complain that management bought a useless sytem.

But then, do this:

Write a tiny javascript that
1) opens a HEAD XML request to the ticketing system page. Here are instructions: http://www.jibbering.com/2002/4/httprequest.html
2) saves the last modified date
3) waits a minute
4) repeats step 1, compares the results of step 2 to the saved result, and redirects to the ticket page if the the date is not the same as the saved one, otherwise repeat.

Put the javascript into an html page, load the html page and bookmark it, and wait. When it sends you to the ticket page, do whatever you do, then reload your javascripting page with the bookmark.

For fifty bucks, I'll write that, but you can get it done more cheaply on some code bidding site, or you can do it yourself and feel the joy of craftsmanship.
posted by orthogonality at 2:25 PM on January 13, 2009

Yeah, step 1 is to post the name of the ticketing system since one of us has likely used it before and/or can google to find the settings to change.

Other than that, here's a shellish method:

#!/usr/bin/env bash
url = "http://URL"
checkfile = "~/pagecheck.md5"
email  = "[e-mail]"
savefile = "/dev/null"
[ -f ./pagecheck ] \
  && \
  (diff $(wget -O $savefile $url | md5sum) $checkfile) && mail -s "UPDATED: $url" $email) \
  || \
  (wget -O $savefile $url | md5sum > $checkfile)

posted by rhizome at 2:44 PM on January 13, 2009

I wrote a stupid, simple little script that checks a given URL continuously until it changes, at which time it displays a popup and quits. It's sloppy, but it's written in AutoHotkey, so I'll bet you'll be able to update and modify it on your own if you so desire. Took me five minutes. Here's a complete .exe that's relatively independent and will work on just about any modern Windows machine, and here's the source-code that you can tweak and re-compile if you choose. Check out AutoHotkey to do it if you wish - I found it a great way to start script-writing.

Hope that's some help. On preview, orthogonality's way is probably a bit cleaner.
posted by koeselitz at 3:52 PM on January 13, 2009

Response by poster: Thanks for the responses so far. The product is sold by Viryanet and it is deployed at a large independent telephone company; and you don't want get me started about how it has been deployed and used over the last year.

I guess I'm looking for a solution that will work with Windows 2000/XP and that I can share with my coworkers. A lot of our PCs are locked down but I can probably get that changed if I can demonstrate a good solution.

I've been using portable Firefox and an add-on called ReloadEvery to keep refreshing the screen, but I think our IT department would start screaming if they found out that fifty of us were using it.
posted by someonesomewhere at 5:21 PM on January 13, 2009

They might well scream, but if fifty of you are doing what they're screaming about, and it's for a clearly work-related reason, and you're only doing it because what they gave you is unworkable, their screams will come to nothing and you win. If they scream about just you doing it, you may lose.

This is one of the very few times I've ever recommended that somebody defy their corporate IT people. But having worked both sides of the IT department fence, it seems likely to me that your is clueless. Firefox away.

Also, it seems to me that orthogonality's approach ought to work as a bookmarklet, and might therefore not even require you to find a place to host a web page.
posted by flabdablet at 6:42 PM on January 13, 2009

I would just write a script that periodically does a wget and then either greps for tickets or diffs against the previous reply, or somesuch trick.

This is easy, even a novice can do it.
posted by rr at 7:07 PM on January 13, 2009

koeselitz writes "Hope that's some help. On preview, orthogonality's way is probably a bit cleaner."

Yeah, I'd go the extra mile and write it for him like you did koeselitz, but the last time I did that for an askMefi, I didn't even get a thank-you for it.
posted by orthogonality at 7:54 PM on January 13, 2009

Use WebMon. It is a very simple (and free) little program that runs in your taskbar, quietly checking for updates on pages you specify (at the interval you specify). When the page updates, you'll get an alert noise, a message will slide up from your taskbar and the icon will change.

The program requires installation, so you'll probably need admin privileges to install.

I use this program, and I love it. It's easy to use, and has some nice advanced features if you like to tinker.
posted by upplepop at 8:19 PM on January 13, 2009

someonesomewhere: I've been using portable Firefox and an add-on called ReloadEvery to keep refreshing the screen, but I think our IT department would start screaming if they found out that fifty of us were using it.

Jesus, you've got an IT department to get annoyed about such things? What a weird situation, although it's all too common; as evinced by the reaction here, any IT person worth her/his salt would get infernally annoyed at a system that works as badly as yours seems to; yet management doesn't seem interested in even letting IT know how it's working.

My advice: befriend somebody who's in IT. Pull them aside and chat for a few moments. Bring this up and mention that, off the record, the only way to deal with it is through a Firefox plugin... "but, friend, we both know that's not a very good solution, and I'd rather not be doing it." Ask for advice; my guess is that it'd take almost no time for anybody with admin privileges to fix this.

If management is doing nothing, then you have to go straight to IT. It won't be a significant amount of work for them, trust me, and they'll almost certainly prefer fixing this for you to the alternative of having extraneous software installed covertly all over the place. And if you and IT work it out swiftly and easily between yourselves without any headache, well, management can't be anything but pleased, right?
posted by koeselitz at 7:50 AM on January 14, 2009