Comments on: American English word frequencies?

Question: American English word frequencies?

aparrish — Mon, 09 Apr 2007 21:38:40 -0800

Where can I find a good word frequency list for American English?

Requirements: Not just sorted by frequency, but with specific frequency information for each word. Not lemmatized. These lists are almost perfect, except they're based on British English (and they have separate entries for "n't" and "'s").

Does a such a list even exist? Is this the kind of thing I can't get online? Google has revealed to me only other lists based on the British National Corpus and a lot of word lists for open source spell checkers and Scrabble programs. Am I missing something?

(Suggestions for other interesting corpora that I could use to generate my own list of this sort would also be appreciated—this is for a generative poetry project.)

By: demiurge

demiurge — Mon, 09 Apr 2007 22:01:18 -0800

I suggest training your generator on a specific poet and seeing if you can get it to emulate that style. Then you can add in other base text to get more breadth.

For example, generate frequencies from http://www.infomotions.com/etexts/literature/english/1600-1699/shakespeare-sonnets-59.txt

By: epugachev

epugachev — Mon, 09 Apr 2007 22:16:29 -0800

The Brown Corpus was what they referred us to in a computational linguistics course I took ~5 years ago. It seems like there may be copyright issues that explain why it is not more widely available, but the first page of google results includes this plaintext version.

The LDC at Penn may be of help.

By: demiurge

demiurge — Mon, 09 Apr 2007 22:21:44 -0800

Oh, of course if you generate frequencies yourself, you'll probably need a part-of-speech tagger, which could be extra work for you.

I'm not sure I understand what's wrong with the lists you linked to, do you just not want British English?

By: lukemeister

lukemeister — Mon, 09 Apr 2007 22:28:43 -0800

The first release of the American National Corpus is available for $75 for non-commercial use.

By: ALongDecember

ALongDecember — Mon, 09 Apr 2007 22:29:40 -0800

The MRC Psycholinguistic Database has helped me before for psychology projects. It runs a Unix dict command on the database based on parameters you set, and it includes Brown Frequency. Take a look at the parameters, to get a list of the most popular words set the min on the BROWN-FREQ at about 50 or so and you'll get a large enough list. You'll have to sort the list in Excel, but it'll cut and paste easily.

By: UbuRoivas

UbuRoivas — Mon, 09 Apr 2007 23:15:35 -0800

wordcount?

i think they have an API as well, if you are technically-minded.

By: UbuRoivas

UbuRoivas — Mon, 09 Apr 2007 23:17:53 -0800

oh, cancel that.

WordCount data currently comes from the British National Corpus.

By: Citizen Premier

Citizen Premier — Tue, 10 Apr 2007 00:44:44 -0800

Do the wiktionary:Frequency Lists help?

By: miagaille

miagaille — Tue, 10 Apr 2007 07:05:52 -0800

Seconding the MRC database. However, please note that the Brown-Freq it gives is NOT THE SAME THING as the Brown Corpus, which is probably what you want. The labels are by editor, so the data for the Brown Corpus is under Kucera-Francis Frequency (edited by Francis and Kucera, 1967). "Brown" here refers to the London-Lund corpus (edited by Brown, 1984). The LLC corpus is British English, the Brown is American.

By: snownoid

snownoid — Tue, 10 Apr 2007 09:07:02 -0800

The nltk-lite corpora package includes a pos-tagged version of the Brown corpus.

By: aparrish

aparrish — Thu, 12 Apr 2007 19:43:13 -0800

Thanks for the leads, everyone!

By: eritain

eritain — Tue, 17 Apr 2007 21:23:29 -0800

The Brown corpus, be it noted, is from the 1960s, and I think it's mostly business/light-literary English. If that's what you want to generate, OK, but the vocabulary might not be very interesting for poetry.

For creative purposes, I think you'd be OK using British sources. If it really had to taste American, you could post-process it for spelling and have/had gotten and so forth. Yes, there are still differences of frequency, but I'm not sure anyone is going to stand over your generator long enough to establish for sure that its ratio of sentence-initial so to sentence-final then is too low for red-blooded American poetry.

And for what it's worth, I once used Variation In English Words (a web interface to the British National Corpus) to search for words that were exceptionally more common in one register than another. For example, I took the adjectives that are relatively prevalent in fiction as contrasted with news; the top five hits were faint, silken, husky, rueful, and momentary—now just try and tell me that isn't Distilled Essence of Bodice-Ripper. I can see those sorts of lists helping you create a near-parodic, over-the-top feeling—and that, for me, is what makes generated art work.

Or you could slurp the collected works of Emily Dickinson from Project Gutenberg, and Perl up the frequency lists yourself. Possibilities ...