July 19, 2004

5:45 AM

I want to update the MetaFilter: The Taglines page, but really really don't feel like doing it the hard way again. It occured to me that it would probably be easy to write a script to search the database for me and load the relevant bits into the database that holds the taglines page (SQL/MT). There is only one problem with that: I have no idea how to go about this. [more inside]
posted by dg to (2 comments total)

Originally, I did this by searching for strings like MetaFilter: and a few variants, then pasting various items into the right places, which is slow and tedious. It seems that it would be simple for a script to look for specific strings, then collect the relevant data around the string and put each bit in the proper field in the database. This brings a couple of questions:

1. Is this possible?
2. If it is not possible to automate the export of the data to the MT site database, is it possible to export it to (for example) a text file, which would mean manually importing the data, but would eliminate the tedious searching?
3. If it is possible, would someone like to write it for me? I could pay a small amount in return, or perhaps provide some hosting or something.

I guess it would need two variants of the script - one to go back to where I left off doing things manually and bring it up to date, then one that would be run regularly (cron job perhaps?) that would create the MT posts and leave the post status as draft so that they could be checked manually.

It doesn's sound too hard, but I am aware that seemingly simple things are often the hardest. Any suggestions would be welcome.
posted by dg at 5:45 AM on July 19, 2004

Well, y6y6y6 just wrote a script to scrape the whole of ask mefi into a database (see the metatalk thread), so it probably wouldn't be too hard for him to adapt that to scrape mefi proper and do what you're after. You should ask him about it.
posted by reklaw at 11:29 AM on July 19, 2004