Ask MetaFilter posts tagged with scraping

Web scraping for dummies

Foci for Analysis — Wed, 06 Aug 2008 15:06:05 -0800

How does web scraping work with PHP/mySQL? What best practices are there? I'm curious about how price comparison services do and manage web scraping, i.e. finding information in unstructured HTML files over many different sites and presenting the information on their own sites. Ultimately, I would like to learn enough about web scraping so that I can create a functional site that, for example, displays a list of dishes that are linked to various recipe sites.

Stuff that I wonder about:
1. In general terms, how would you code the project using PHP/mySQL? Any code libraries that can be used for scraping?

2. I understand that you can regexp data from the scraped html files, but aren't there more intelligent ways of extracting the data? I'm thinking about XSLT and such.

3. How do you handle form generated pages? For example, recipe sites that allow you to search form recipe by using check boxes, pull down menus, etc? Again, are there any smart code libraries out there that simplifies this?

4. Are there any best practices regarding managing scraping, storage, data manipulation, performance, ethics, etc, that I should be aware of?

Gathering ebay infos.

SirStan — Wed, 23 Jul 2008 15:48:29 -0800

cp /mnt/com/ebay/completed_auctions/ps3 ~/ps3.txt ?? I have the following pieces of information:

a. Search criteria
b. Date range.

And from this; I would like an Excel spreadsheet that has the following columns:

end price, end date, description, item #, seller, etc

Is there any way to do such a thing in an automated fashion without screen scraping myself?

Datamining the public web

ilike — Mon, 30 Jul 2007 01:52:51 -0800

How do i build a data warehouse that scrapes data from public websites for my own use? Tools? Tips? Hi. I would like to track apartments on a classifieds site and use the data for analyzing the inpact of diffrent things on price. What i need is a tool or scripting language that would make it easy for me to spider the website and put the data in a database. Preferable this would be an open source solution.

I am also looking for good tools for extracting information out of longer pieces of text. For example on the site i want to mine users can put in comments on every object. I would like to be able to decide if a comment is positive, negative och neither. I have seen this be done on one online art site that i cant remember the name of right now. The artist used blog post and decided the mood of the writer by what words were used.

What is that noise?

poweredbybeard — Fri, 08 Jun 2007 09:52:00 -0800

Bicyclefilter: A pronounced grinding/scraping from my rear wheel. Is this potentially unsafe, or simply embarrassing, to ride? When I say pronounced, I mean passersby chuckle. The wheel sounds really old, really sad and really grumpy. When I spin the wheel freely, the gears wobble a bit, leading me to think the threads on the gears or the axle are close to stripped (?? I can still pedal just fine, so I don't know).

What's the worst that can happen, realistically? The wheel falls apart? Or I just suddenly find I'm pedalling but not moving?

Web scraping onling banking sites?

mphuie — Thu, 07 Sep 2006 16:28:36 -0800

Is there an tool/tutorial to web scrape secure sites like Online Banking or other sites that make sure that it's you that is logging in? I've been self learning web scraping and found it extremely useful, building proxies for my Nokia tablet, automation, through Perl & LWP.

I want to write something that will grab my bank balance, but those online banking sites seem to go through lots of hoops to make sure you are the one sitting at your computer trying to login.

I've tried deciphering all the tactics they use, javascripts, cookies, but it seems like theres more tricks they are using that I don't know about.

Is there some util that lets you analyze the actual behind the scenes events, headers, cookies, etc.. so I can just repeat them?

A low level tool would be great, but I'd be happy with a web recorder/automation program that would log me in (based on keystrokes, etc) and save the html so I could get my balance.

Image search engine scraper/downloader

Moistener — Tue, 16 May 2006 22:08:34 -0800

I need a web scraper script/proggie to download and thumbnailize images for a long list of search terms. .

Input: Long list of search terms.

Output: corresponding images and thumbnails on my hard drive, arranged/accessed/organized by search term.

My already feeble google-fough (<-- note ignorance of even the spelling) is failing me.

Web Scraping for dummies?

Kwantsar — Fri, 22 Jul 2005 11:37:48 -0800

A project at work has come up, and I would save a lot of time and hassle if I could somehow get my hands on a free (cheap is acceptable, as long as I can try it first), easy to use web-scraping program. The URL from which I will be scraping is static, unencrypted, and otherwise extremely vanilla. Suggestions?

real estate hacks: scraping together a house

craniac — Wed, 13 Jul 2005 08:11:38 -0800

I'm looking for a home closer to my work, and I'm trying to be systematic about it. My plan is to scrape online real estate databases and put them into a spreadsheet, then start ranking and comparing. So, I guess I have two questions:

1. What would be the best tool (text wrangler?) to extract fields from an archived html page? (I am using OS X Tiger and am loosely familiar with regex)

2. Would I probably be better off just driving around key neighborhoods and jotting stuff down by hand?

(Sample database: Utah real estate)

Question number 4865

PrinceValium — Fri, 23 Jan 2004 17:02:47 -0800

Is there a way, without constant human intervention, to (1) mine either Google News, Yahoo News, or the AP for new obituaries and (2) drop the name, age, blurb, and URL into a database?