Show me the Links!

September 16, 2007 6:14 PM

Looking to parse an RSS feed and pull all the links out, and then turn links into a new feed.

I've search around stumbled across magpie, simplepie and carp but not sure if those solutions will perform the actions I'm looking for. Does this seem like a custom script or has anyone heard of any parsers out there that will perform this action?

Thanks!!

posted by bleucube to Computers & Internet (6 answers total)

Yahoo! Pipes!
posted by furtive at 6:32 PM on September 16, 2007

Not sure what you mean here. I'll take a guess.

Do you want to parse the content elements of an RSS feed, find the html links and make an RSS out of those html links?
posted by sexymofo at 6:42 PM on September 16, 2007

If you know python the combination of ScrapeNFeed and (the superb) BeautifulSoup lets you knock stuff like this up in minutes.

ScrapeNFeed makes it easy to generate RSS feeds, BeautifulSoup makes it easy to pull stuff out of HTML/XML.
posted by markr at 12:43 AM on September 17, 2007

Response by poster: SexyMofo - Yep that's what I'm looking to do.

Markr - Not familar with Python, but doesn't hurt to expand knowledge, appreciate the tip.
posted by bleucube at 5:13 AM on September 17, 2007

As markr points out, the hard part is parsing the content items since it's likely not to be well-formed. If you're more familiar with java, you could use something like TagSoup or HTMLCleaner.

Parsing the RSS feed (using something like Rome), scrubbing the content, getting the links and producing the new feed would be fairly trivial, or at least straight-forward.

So the tools are there (either in java, PHP or python)--I'm just not sure how to plug into a service like Yahoo Pipes.
posted by sexymofo at 5:43 AM on September 17, 2007

Response by poster: Thanks! Will give me enough to go on. Will provide update in a bit. This is my number 1 project at the moment.
posted by bleucube at 6:40 AM on September 17, 2007