real estate hacks: scraping together a house

July 13, 2005 8:11 AM

I'm looking for a home closer to my work, and I'm trying to be systematic about it. My plan is to scrape online real estate databases and put them into a spreadsheet, then start ranking and comparing.

So, I guess I have two questions:

1. What would be the best tool (text wrangler?) to extract fields from an archived html page? (I am using OS X Tiger and am loosely familiar with regex)

2. Would I probably be better off just driving around key neighborhoods and jotting stuff down by hand?

(Sample database: Utah real estate)

posted by craniac to Home & Garden (8 answers total)

The best strategy in my experience is get a feel for the community you'd like to live in (by driving around), and use a good realtor to do the leg work for you. They will let you know the instant something comes up in the area you are looking into.
posted by qwip at 8:19 AM on July 13, 2005

Find a multiple listing real estate site (mls) for Utah. These are pretty common and most real estate agencies even provide them now. This will provide listings from all agencies, without having to extract them from each website. Realtor.com, for example, is a nationwide one.
posted by undertone at 8:23 AM on July 13, 2005

When we did the house-search, we printed out the MLS entries and stuck them in a folder. Folder and people in car. We then looked at the neighborhoods, the houses themselves, and whatever we could see from the outside. Jot notes and questions on MLS pages. Sort pages into "we want to see this with our agent" and "we never want to see again." Further sort into "Really Think We Like" and "Lukewarm." Visit houses with agent. Repeat as needed.
posted by Medieval Maven at 8:44 AM on July 13, 2005

I have to agree with quip. Research the market so you can assess property and value quickly, but then find a good buyer's agent familiar with the neighborhood you want to be in. Many times, the best places get sold during the realtor's open house (before the public is even aware of the listing). I got my first condo that way...my agent called me from the open house he was attending for realors, I left work and slipped in with the other realtors and made my offer. Even so, I was competing against two other offers and it had only been on the market for a matter of hours! When we sold that condo, it sold in 3 hours after the realtor's open house started...at full price. Our next house never went on the market, we knew about it through neighbors in the neighborhood we wanted to be in. Best of luck...
posted by jeanmari at 8:44 AM on July 13, 2005

A service that I used in buying my last house was Realtor.com. Lots of criteria by which you can narrow your search. In fact, in selling my last home (in Stamford, CT -- a very volitile market) I was able to use this database to run an ad-hoc comparitive market analysis to reprice my home within reasonable boundaries.
posted by thanotopsis at 9:42 AM on July 13, 2005

I did exactly what you want, when house shopping last year.

I used Python and "beautifulSoup" to scrape the HTML. The program wasn't more than twenty lines long. All it did was grab the MLS listings, scrape the MLS numbers, identify the new ones, and then pickle the database for next time 'round. The new MLS listings were opened in my browser.

A word of warning, though: it's illegal to scrape MLS. The organization won a lawsuit against a realty company that was scraping the database. Proceed with caution. IMO, if you are running it only once a day, for non-commercial use, and not sharing the information, you can probably get away with it. IANAL, and you might get nailed.
posted by five fresh fish at 10:05 AM on July 13, 2005

I used ZipRealty to do exactly this. They give users the ability to store several arbitrary locations, and then houses every house you examine will be labelled to show its distance from those locations. I used this setup to find houses within a mile of the DC subway system, the Metro.

basic steps:

1. Create an account at ZipRealty
2. Once you're logged in on your account, enter your work address as one of "My Places" via this link.
3. Create and save an MLS search by zip code(s) to return the homes in the general area of interest.
4. Examine the homes returned by your search for distance to "Your Places"

ZipRealty is free, and using them doesn't create any obligations (I didn't use them for anything but the searches). I'd think twice about giving them valid contact info, though, for privacy's sake.
posted by NortonDC at 11:24 AM on July 13, 2005

Canadian Real Estate
posted by blue_beetle at 11:38 AM on July 13, 2005