Ask MetaFilter questions tagged with postgres

How does flickr do it?

hayvac — Wed, 04 Feb 2009 18:18:38 -0800

My developer and I want to create a database schema that allows us to search through hundreds of tags applied to thousands of records. So far our benchmarks suck. Are we expecting too much? Here's the deal (and I'm not the database guy, so bear with me if I mess up the nomenclature): We have a bunch of records and want to apply tags to them. We're testing a database schema with dummy data including 100,000 records and 750 tags, with an average tag-to-record count of 100. This doesn't perfectly approximate our actual data set, but it's close enough.

So we have Table 1, which lists the tag ids, Table 2, which lists the record ids, and a join table, which is a two-column table listing a tag id in one column and a record id in the other wherever that tag is associated with that record.

For example, if tag223 is associated with record12, record6994, and record90001, the join table looks like this:

tag223 | record12

tag223 | record6994

tag223 | record90001

(Given the numbers above, the actual join table has around 10,000,000 rows.)

This seemed to make sense when he explained it to me, but the benchmarks with the dummy data, even indexed, were ridiculously slow. This was just one a desktop machine, but it was several orders of magnitude slower than we want it to be.

Is it possible to do tag-based searching on this scale with decent performance? Are we missing something simple?

We're using Postgres. The final product is going to be extremely read-heavy (i.e., once the data's in it'll be only minor updates). If I've left anything out, please ask.

Did the earth move for you too?

genghis — Sat, 10 Jan 2009 12:24:08 -0800

GISfilter -- I have a PostGIS table containing spatial data in the form of a few thousand shapefiles. Anybody know a quick way I can move *all* of them by about 40 metres? Due to a bizarre set of circumstances I need to work with a data set of things contructed from GPS in Ireland. Because of a systemic error, every multipolygon I have is out of whack slightly. Boo!

Yet, because the country I live in is small enough for this to work, I can have a bunch of stuff accurate to within 2m if I globally move every multipolygon 50m to the North and 23.4m to the West. Yay!

But I'm too unfamiliar with both Postgres and Python (I'm working with Geodjango for the first time) to figure out which 10-line loop script will do that lateral translation for me. Boo!

So... um... can anybody lend me a hand here? I don't really care which language the one-use-ever script I need is written in. And my multipolygons are strings of co-ords using WGS84, where each X&Y value is separated by a space, and each pair separated by a comma.

The application of your collective genius is warmly appreciated.

MySQL vs PostgreSQL?

xmutex — Thu, 22 May 2008 16:32:05 -0800

What's the hive mind's opinion on MySQL vs PostgreSQL? In a web dev shop, rebuilding the platform. It seems like at MySQL 5.2, the differences have become rather negligible, but I am still interested in your experiences. Currently we're using MySQL (5.0 though), but I've always heard wonderful things about Postgres's robustness and speed.

Everything I find on google for 'mysql vs postgresql' seems to date bate to 2003.

Converting Postrges-specific query to Mysql

mcstayinskool — Fri, 14 Jul 2006 09:19:40 -0800

I need help converting a Postgres-specific SQL query to MySQL I'm in the middle of converting an application to use a Mysql back-end instead of Postgres. The database schema I got converted fairly easily, but one of the SQL queries that the app executes is failing, and I don't know how to write it to make MySQL happy. I'm assuming that the EXTRACT(epoch from task_id::abstime) is where it's failing.

Anyone with good Mysql-fu that can help?

SELECT EXTRACT (epoch from task_id::abstime),
task_name
FROM tasks WHERE task_id < now()
AND task_state = '%d';