Ask MetaFilter questions tagged with scraping

Scraping & saving?

extropy — Thu, 19 Nov 2009 15:13:41 -0800

I have perhaps a thousand delicious links (to documents in the SEC database). All of these could be broken at anytime if the SEC changes the way it displays these. How do I automate the process of copying the contents of those documents so I can save them in a database? I have checked into previous questions and web scraping software, but the web scraping/crawling/spidering software out there requires what looks a little to much to me like programming.

Is there an easy way to collect the documents? I am hoping to feed something the list of links and be done. Fair warning: if I can figure this out, then I will ask how best to save the documents in a database. I have considered using Mechanical Turk, or something, but I think this really ought to be a job for a machine. Free software solutions preferred, but willing to pay to make it easy for me to do...

Sample document:

Not-so-sof(f)t shoes

pxe2000 — Thu, 24 Sep 2009 06:37:01 -0800

Shoe problems. How do I keep a new-ish pair of shoes from scraping my heels/ankles? I recently bought a pair of kitten-heeled Sofft pumps at a thrift store. They are adorable and fit me perfectly, except for one problem: The back of the shoe scrapes against my heel when I walk. I assumed this would stop after I've broken them in, but after I wore them out for two full days they persisted in scraping against my heels.

I recently had to purchase a few new pairs of shoes, and had similar problems during the break-in period. The blisters from those are still present, and breaking in YET ANOTHER new pair of shoes may be exacerbating an existing problem. Would stretching the heels help? (When I bought them, the heels were a bit narrow, as though the owner kicked them off by stepping out of one with the opposite foot.)

I don't work for the government, promise.

McBearclaw — Thu, 11 Jun 2009 18:32:32 -0800

I'm trying to map protests in the United States, but I'm grappling with data sources (and will eventually tangle with data management). Any ideas? I'd like to map out protests, riots, bombings, and other cheerful social outings - ideally in the United States, where I have the most contextual knowledge, but that's not a necessity.

My original plan was to scrape AP's US news RSS feed, store everything in some sort of XML database, and then query that for what I need. I just checked their RSS format, and it unfortunately doesn't include the full article. Nor does it include a separate tag for the location, which would make geocoding a bit/much nastier. NYT's feeds are basically the same story. I don't really know where to go from here.

There are basically five steps, and I would love advice on any:
1. Scrape database of news articles.
2. Store in a format that would allow querying by date or location. I'd like to keep all the articles, too, because... really, that would be an awesome dataset.
3. Tag protests (method: NLP, Mech Turk, or caffeinated McB).
4. Tag with date and location.
5. Make pretty maps.

Step 6 is going crazy with spatial stats, but I've got that part covered. I've been letting this project fester for too long, and it is now certifiably brain crack. Any advice on 1-5 would be greatly appreciated.

Aside: I really have thought about the ethical consequences of this. If you're concerned, MeFiMail me and I'll do my best to assuage your doubts.

Scraping paint.

OmieWise — Thu, 07 May 2009 16:17:43 -0800

How will I know when I'm done scraping the paint off of this window molding? What are the final steps before painting? I've been using a combination of a stripper and scraping to remove paint from a reasonably ornate window molding. Some areas are bare wood, some are close, and some still have several layers of paint on them. Do I need to go to bare wood all over, or is some paint left ok as long as there aren't abrupt level transitions between areas?

My assumption has been that the final step will be to use a coarse grit paper to get rid of any leftover paint nubbins. What is the best way to clean off all the paint dust after that? Should I be doing something else?

The whole will be repainted with latex primer and enamel.

Newspaper Clippings 2.0?

[insert clever name here] — Thu, 07 May 2009 15:36:38 -0800

Can I automate archiving/saving news articles on a certain topic I pull from google new's rss feed? For a while, I was manually copying and saving all news articles from google on a certain topic I got in my rss feed. But it became cumbersome so I stopped. Now I've looked back at those articles from a few years ago, and wish I had kept up. Is there a way to automate something like that? A modern day newspaper clipping collection only automated? I don't want to save just the url, but the actual text of the article, where it is from, date, and possibly pictures.

This is for my own personal use, so I doubt it would fall under any copyright issues (I would assume).

I did a search but my google fu is failing me. I keep coming up on the google news archive, but thats not really what I'm looking for. I want my own personal copies. I don't now how google news archive works, but I know that some articles I had gotten from google news originally are not in their archive (I just checked.).

Running into Ruby Restriction

FuManchu — Mon, 09 Mar 2009 04:06:27 -0800

In Ruby, how can you get around the ~65,500 character limit when grabbing a web page? I'm new to Ruby, but have started to use it to scrape information from websites. I have been using either the Hpricot package or net:http. Unfortunately, when I've tried using these to scrape larger web pages, the streams cut off after 65,500 characters. I haven't found any information online about this. Is there a way to get around this limit? Can you separate the stream over two arrays or strings? Or will I have to manage the stream myself with new code?

Web scraping for dummies

Foci for Analysis — Wed, 06 Aug 2008 15:06:05 -0800

How does web scraping work with PHP/mySQL? What best practices are there? I'm curious about how price comparison services do and manage web scraping, i.e. finding information in unstructured HTML files over many different sites and presenting the information on their own sites. Ultimately, I would like to learn enough about web scraping so that I can create a functional site that, for example, displays a list of dishes that are linked to various recipe sites.

Stuff that I wonder about:
1. In general terms, how would you code the project using PHP/mySQL? Any code libraries that can be used for scraping?

2. I understand that you can regexp data from the scraped html files, but aren't there more intelligent ways of extracting the data? I'm thinking about XSLT and such.

3. How do you handle form generated pages? For example, recipe sites that allow you to search form recipe by using check boxes, pull down menus, etc? Again, are there any smart code libraries out there that simplifies this?

4. Are there any best practices regarding managing scraping, storage, data manipulation, performance, ethics, etc, that I should be aware of?

Gathering ebay infos.

SirStan — Wed, 23 Jul 2008 15:48:29 -0800

cp /mnt/com/ebay/completed_auctions/ps3 ~/ps3.txt ?? I have the following pieces of information:

a. Search criteria
b. Date range.

And from this; I would like an Excel spreadsheet that has the following columns:

end price, end date, description, item #, seller, etc

Is there any way to do such a thing in an automated fashion without screen scraping myself?

Datamining the public web

ilike — Mon, 30 Jul 2007 01:52:51 -0800

How do i build a data warehouse that scrapes data from public websites for my own use? Tools? Tips? Hi. I would like to track apartments on a classifieds site and use the data for analyzing the inpact of diffrent things on price. What i need is a tool or scripting language that would make it easy for me to spider the website and put the data in a database. Preferable this would be an open source solution.

I am also looking for good tools for extracting information out of longer pieces of text. For example on the site i want to mine users can put in comments on every object. I would like to be able to decide if a comment is positive, negative och neither. I have seen this be done on one online art site that i cant remember the name of right now. The artist used blog post and decided the mood of the writer by what words were used.

What is that noise?

poweredbybeard — Fri, 08 Jun 2007 09:52:00 -0800

Bicyclefilter: A pronounced grinding/scraping from my rear wheel. Is this potentially unsafe, or simply embarrassing, to ride? When I say pronounced, I mean passersby chuckle. The wheel sounds really old, really sad and really grumpy. When I spin the wheel freely, the gears wobble a bit, leading me to think the threads on the gears or the axle are close to stripped (?? I can still pedal just fine, so I don't know).

What's the worst that can happen, realistically? The wheel falls apart? Or I just suddenly find I'm pedalling but not moving?

Web scraping onling banking sites?

mphuie — Thu, 07 Sep 2006 16:28:36 -0800

Is there an tool/tutorial to web scrape secure sites like Online Banking or other sites that make sure that it's you that is logging in? I've been self learning web scraping and found it extremely useful, building proxies for my Nokia tablet, automation, through Perl & LWP.

I want to write something that will grab my bank balance, but those online banking sites seem to go through lots of hoops to make sure you are the one sitting at your computer trying to login.

I've tried deciphering all the tactics they use, javascripts, cookies, but it seems like theres more tricks they are using that I don't know about.

Is there some util that lets you analyze the actual behind the scenes events, headers, cookies, etc.. so I can just repeat them?

A low level tool would be great, but I'd be happy with a web recorder/automation program that would log me in (based on keystrokes, etc) and save the html so I could get my balance.

Image search engine scraper/downloader

Moistener — Tue, 16 May 2006 22:08:34 -0800

I need a web scraper script/proggie to download and thumbnailize images for a long list of search terms. .

Input: Long list of search terms.

Output: corresponding images and thumbnails on my hard drive, arranged/accessed/organized by search term.

My already feeble google-fough (<-- note ignorance of even the spelling) is failing me.

Web Scraping for dummies?

Kwantsar — Fri, 22 Jul 2005 11:37:48 -0800

A project at work has come up, and I would save a lot of time and hassle if I could somehow get my hands on a free (cheap is acceptable, as long as I can try it first), easy to use web-scraping program. The URL from which I will be scraping is static, unencrypted, and otherwise extremely vanilla. Suggestions?

real estate hacks: scraping together a house

craniac — Wed, 13 Jul 2005 08:11:38 -0800

I'm looking for a home closer to my work, and I'm trying to be systematic about it. My plan is to scrape online real estate databases and put them into a spreadsheet, then start ranking and comparing. So, I guess I have two questions:

1. What would be the best tool (text wrangler?) to extract fields from an archived html page? (I am using OS X Tiger and am loosely familiar with regex)

2. Would I probably be better off just driving around key neighborhoods and jotting stuff down by hand?

(Sample database: Utah real estate)

Mining news sites for data.

PrinceValium — Fri, 23 Jan 2004 17:02:47 -0800

Is there a way, without constant human intervention, to (1) mine either Google News, Yahoo News, or the AP for new obituaries and (2) drop the name, age, blurb, and URL into a database? I've pondered this for a while. A really crude way would be to search headlines for ", [0-9][0-9], " and " dies at [0-9][0-9]." But I'm not sure this would pick up everything. For example, if I search Google News for "kangaroo" I get only two links out of about 20 that identify Bob Keeshan's name, the reason for his fame, and his age. Most say simply "Captain Kangaroo Dies". And only the NYT headline has all the data elements separated by commas (and is likely not consistent on that point with each obit.)

Any cleaner ideas?