Comments on: How to extract data from HTML

Question: How to extract data from HTML

elgilito — Wed, 22 Apr 2009 15:17:35 -0800

How to extract data automatically from a group of HTML pages to an Excel file or to a database?

I have a large group of old HTML files (around 800) that are more or less organised the same way. I need to extract certain items (the title, the h2 and h3 headers etc.) to store them in a database. The long term goal is to replace the static pages by dynamic ones but first I need to know what's in there without looking at each file, and I'll have to correct, reassign and rewrite some of the content anyway, so I need to have everything in a easy to browse format. I can write a VBA script (for Excel or Access) or a PHP script (for Mysql) but I was wondering if there were a simple, free tool for this (for Windows). If the tool could take the tag type (h2) and the file directory and spit out a CSV file with "filename, tag content" that would be enough for me.

By: wongcorgi

wongcorgi — Wed, 22 Apr 2009 15:33:41 -0800

I don't believe there are any tools this specific, or at least will batch process all your files and export to the format that you want.

Since you already know PHP, look into the regular expressions with the preg_match/preg_match_all functions.

By: rokusan

rokusan — Wed, 22 Apr 2009 17:22:11 -0800

The Python library Beautiful Soup is made for this kind of thing.

My Python skills are poor, but I'm learnin', and I managed to find it, get it installed, write a script and churn through 4000 HTML pages in a single afternoon. I wanted tab-delimited output, which is pretty close to what you want.

By: signal

signal — Wed, 22 Apr 2009 21:19:49 -0800

Yeah, Beautiful Soup is made for this.

By: elgilito

elgilito — Thu, 23 Apr 2009 04:47:08 -0800

Thanks for the tip about Beautiful Soup. I had trouble figuring out how to install it but I'm running a script now and it's working well!
(I've used regular expressions in PHP, but I'm doing this very occasionally so every time I have to spend time figure them out again).

By: teabag

teabag — Thu, 23 Apr 2009 05:38:33 -0800

You can also use CURL + SED/AWK shell scripts, or CURL with Perl, or Curl with Python.

That Beautiful Soup thing is interesting, never seen it before. Gonna have to try it out.