Comments on: How can I grab the text (not code) off of a bunch of .htm files?

Question: How can I grab the text (not code) off of a bunch of .htm files?

pantufla — Tue, 13 Feb 2007 18:25:36 -0800

How can I automatically grab the text (not code) off of a bunch of .htm files?

I have a bunch of .htm files which are based on the same template, and I am looking for a way to grab all the text from these pages and collect it in a text file for a voice actor to read. I could copy each page's text through a browser but I thought there had to be an easier way, as I need to grab the text from over 100 pages. Any advice appreciated!

By: saraswati

saraswati — Tue, 13 Feb 2007 18:37:07 -0800

Here is a tutorial on stripping HTML tags using regular expressions. It includes VBScript examples. If you google around you'll find some simple VBScript code that will load in the files and all you need to do from there is enclose it in a loop to do all of the files.

By: pantufla

pantufla — Tue, 13 Feb 2007 18:37:32 -0800

Thanks qvtght. Is there any perl script I could use that I wouldn't need much knowledge of perl to run?

By: ReiToei

ReiToei — Tue, 13 Feb 2007 18:38:20 -0800

"Perl."

... real helpful.

http://www.webscrape.com/

By: qvtqht

qvtqht — Tue, 13 Feb 2007 18:52:01 -0800

pantufla: Apologies about the terse response.

An easy way a non-programmer could accomplish this is by using a multi-document text editor such as EditPlus. Open all of the HTML files, and then do a global search-and-replace for the common elements you want to replace. You can use regular expressions to strip out all HTML tags by searching for "\<.+\>" (without the quotes).

Other options include:

http://www.velocityscape.com/
http://www.iopus.com/imacros/web-testing.htm
http://www.theeasybee.com/

By: xiojason

xiojason — Tue, 13 Feb 2007 19:08:32 -0800

lynx -dump http://ask.metafilter.com

By: xiojason

xiojason — Tue, 13 Feb 2007 19:12:11 -0800

or, if you made a textfile with the URL of each page on a separate line and have access to a bash shell:

cat urls.txt | while read url; do lynx -dump "$url" >> pages.txt; done

By: 31d1

31d1 — Tue, 13 Feb 2007 19:16:45 -0800

Yeah, something like:
lynx -dump -nolist -width=NUMBER [url]

The -nolist flag disables the list of links it prints at the bottom, and -width=NUMBER lets you wrap on something other than 80 characters.

By: Rhomboid

Rhomboid — Tue, 13 Feb 2007 19:55:01 -0800

Yeah, you don't want to try to remove tags yourself, you want to actually render the page. Use lynx.

Pet peeve: "cat file |" is always extraneous and unnecessary. You can replace this with redirection and save having to actually invoke /bin/cat: while read url ; do ... ; done <urls.txt"

By: xiojason

xiojason — Tue, 13 Feb 2007 20:12:35 -0800

Interesting. Is there some significant downside to using cat? I've used cat and avoided the redirection in an effort to increase readability, linking the read to its input in an easy-to-see left-to-right manner. Otherwise the data source can be so far away from the reader that it seems out of place.

By: xiojason

xiojason — Tue, 13 Feb 2007 20:26:46 -0800

Ah, I see from some further reading that, were I using zsh, it would be truly unnecessary, since zsh supports input redirection before the while. Another reason I should probably start messing about with zsh. Sorry for the continued derail.

By: lunchbox

lunchbox — Tue, 13 Feb 2007 21:24:17 -0800

I've done very similar things before, and html2text has always worked for me. It's available as a Linux package; I'm sure there are versions for Windows and Mac too. If you have Python you can use the Python version.

By: Rhomboid

Rhomboid — Tue, 13 Feb 2007 21:58:54 -0800

xiojason - You're right, the actual wastefulness of invoking cat is very minimal and I suppose it does improve readability if you are used to seeing that idiom. But it's just one of those pet peeves that grate me a small bit every time. I regret derailing the thread for such trivial *nix minutiae now.

By: metaswell

metaswell — Tue, 13 Feb 2007 22:21:24 -0800

HTMSTRIP: Processes and removes embedded HTML commands from Web pages downloaded from the Web.

http://www.erols.com/waynesof/bruce.htm

Its a commandline tool, and I use it routinely when I want to save just the text portions of study material from the web, which is quite often!

VERY flexible, and wildcards are allowed. It will go through an entire directory/folder really fast, and the results are good.

By: ejoey

ejoey — Tue, 13 Feb 2007 22:28:44 -0800

If you are on a Mac you can just print to a PDF, or do any of the good unix-y tips above as well. If you are on a Windows machine, you can use PDF Creator to make a PDF as well.

By: raildr

raildr — Tue, 13 Feb 2007 22:28:49 -0800

just load it in a browser, as Rhomboid says, select all, copy and walah!

By: pantufla

pantufla — Tue, 13 Feb 2007 23:49:59 -0800

thanks for all the great info!