Comments on: XML to MYSQL

Question: XML to MYSQL

alse — Mon, 21 Jan 2008 09:19:03 -0800

Is there a simple way to convert an XML feed into a MYSQL database?

Navicat works great, but I'd like something that runs runs on the server, rather than my PC. I've seen the old thread asking the same question, but it did not provide any working solutions. Thanks for the help!

By: zsazsa

zsazsa — Mon, 21 Jan 2008 09:25:50 -0800

You could use XSLT to transform the XML into SQL statements that you can run on the mysql server.

By: AmbroseChapel

AmbroseChapel — Mon, 21 Jan 2008 19:39:35 -0800

"What's an XML Feed?", that's the question. XML can have completely arbitrary levels of nesting and complexity which isn't necessarily easy to map to a row-and-column database.

Assuming that your data is the sort that's amenable to being put into a database, (perhaps you mean an RSS feed?) zsazsa's suggestion would work.

A cron job chops the data up into "insert" statements, saves them into .sql files, cron job runs the files at regular intervals, something like that?

Otherwise, you're looking at some kind of script, in Perl or PHP most likely.

Happy to try and create a script.

By: alse

alse — Tue, 22 Jan 2008 08:16:15 -0800

I'm looking to parse a feed such as this. I hoped to find something more automatic like Navicat, but it looks like I'll have to learn to program it myself.

By: Skorgu

Skorgu — Tue, 22 Jan 2008 08:23:43 -0800

Yikes, that's not even really xml, the good stuff is inside a CDATA. If all you want is to put the "Volcanic", etc data and the link into a database it's a one-liner in any scripting language.

By: Skorgu

Skorgu — Tue, 22 Jan 2008 08:38:45 -0800

Very brittle example:
curl http://www.google.com/trends/hottrends/atom/hourly | perl -nle 'while(m/class="(\S+)".*href="(.*)"/g){print "INSERT INTO foo $bar, baz$ VALUES $ \"$1\" \"$2\"$;"}'

By: AmbroseChapel

AmbroseChapel — Tue, 22 Jan 2008 18:31:41 -0800

Basically the perl script you want goes something like

use LWP::Simple;

use DBI;

$dbh = DBI->connect("DBI:mysql:database=<foo>",<username>,<password>);

$page = get('<url>');

$sth = $dbh->prepare("insert into <table> values ?,?,?");

while($page =~ /<regular expression which grabs 3 things>/g){

    $sth->execute($1,$2,$3);

}

# strict, warnings, add "or die" messages, exercise for the reader etc.

How are you going to handle duplicates?