Ask MetaFilter questions tagged with databases

Automated mapping software

nushustu — Mon, 14 Dec 2009 09:15:03 -0800

I need an app that would allow many users to input a few bits of data - name and zip code, for instance - and then would map the results of all the individual inputs onto a map. Assume that I have no programming skill. What I hope exists is a template or something that could create a form on a webpage that, once a person enters their contact information, it stores said info in a database and simulatenously adds pertinent location data to a map on the same website. So if John in Cincinnati fills out a form, the website updates the map to include a pushpin with John's name in Cincinnati or (preferably) in a particular zip code in Cincinnati.

The push pins don't have to be in some exact location; I don't need it to be so specific that it includes long/lat data.

Even if it's not dead-easy, I can get someone with more computer skills to do this; I just need to be able to tell my boss what the best options might be.

The meaning of "In the gooey"

ergibson — Wed, 21 Oct 2009 15:59:04 -0800

What does the phrase "in the gooey" mean? A person was introducing me to MySQL and referred to the ability to edit data fields as "working in the gooey." The phrase immediately appealed to me, but before I start using it I want a better understanding of what it means. Are there any tech folk out there who use the phrase that can provide me with some insight?

Retail barcodes for tracking my shopping at home.

yoHighness — Sun, 28 Jun 2009 14:31:01 -0800

How do I get access to UK retail barcodes? I would like to build an application for scanning my shopping at home. I would like to keep track of my shopping for starters; but ideally (and utopically?), I would use database mashups to tell me how much energy was spent producing my food, which countries (dictatures) it comes from, how healthy it is, how ethical, could I have gotten cheaper deals etc. etc.
I am based in the UK. Bonus points for comments on how hard this would be to make (and as a mobile app?), and pointing me to similar projects. Thank you, hivemind!

How will getting treated for bipolar affect my job?

Anonymous — Tue, 16 Jun 2009 18:38:23 -0800

What are the repercussions of seeking treatment for bipolar disorder on my employability? I'm in my mid 20's, in the UK. I am concerned that I have Bipolar Disorder, and am planning on speaking to my GP about it. However, a friend who has the condition has mentioned that I will "be added to a database" and have to "disclose that I'm bipolar at a job interview".

Given the current economic situation, I don't want to do anything that will prevent me from getting/keeping a job. I am currently employed, but after hearing that the information must be disclosed, I am concerned that after telling my superiors, I will be first in line if the company I work for does get into trouble. I realise that discrimination based on disability is illegal, but I'm smart enough to realise that rules have been broken before, and that there are all kinds of reasons to give, other than the real one, for firing someone.

My bipolar, if indeed I do have it, does not affect my job at all. In any case, I seem to have a mild case of something, but I never allow it to affect my work. I've been with my current company for over 6 years, with no problems. However, lots of cutbacks are being made, and I don't wish to raise my head above the parapet. I know that it shouldn't be a problem, but I'm not so green as to think that it won't be.

My question boils down to this: how will seeking/getting treatment for bipolar affect my working life? Personal experiences are welcome.

CMS game backend?

twins named Lugubrious and Salubrious — Tue, 28 Apr 2009 11:08:19 -0800

Can I modify an existing CMS to serve as the backend for a browser game? I'm not sure if I really want to build a database from scratch. If I could conceivably use a CMS like Drupal or some other recommendation to handle all the data related to the game, I'd rather do that instead.

Basically it's a very simple RPG wrapped around a forum. So there would be two parts: activity in the forums would affect the user in the game, i.e. a certain post count might give the user access to a new avatar or something.

The other part may or may not be part of the CMS already. It would involve actually advancing the user's avatars. If I used a very simple mechanic, i.e. clicking on the character adds a random point to intelligence or strength, is there a function in any existing CMS that could mimic this mechanic? A sort of click-counting function or something? The game would actually be more complicated than that of course, but what I'm basically wondering is if I can use a CMS to generate dynamic pages for avatars and track the "NUMBERS MUST GO UP" part of the game. Or if I'd have to code something separately and tack it on to the CMS.

I have no experience with MySQL or PHP whatsoever, but I do have experience setting up and using some simple CMS's (Movable Type, Wordpress, MediaWiki, currently using TikiWiki for another project) and letting them handle the messy stuff. I don't mind learning MySQL/PHP eventually, but I'd like to do it by getting started looking under the hood and tinkering around with CMS's instead of just diving in and learning straightaway.

So is this feasible? Are there other (hopefully simple) ways to build this data-driven site? Feel free to also tell me that there is no way to do this and that I just have to learn PHP/MySQL already.

Help me find some numbers.

jbickers — Thu, 23 Apr 2009 12:05:36 -0800

I'm interested in doing data-based mashups. What are some really cool, deep sources of publicly available data? I realize I'm casting a wide net here, but I'm intrigued by the data mashup, and wonder (a) just how much, and how diverse, publicly available data is out there, and (b) where to find it.

So, just pulling some examples off the top of my head ...

* Obesity rates vs. shopping mall density
* Population vs. bankruptcies
* iPod sales over time vs. weather trends in a specific geographic area

... you get the idea. Anything goes. Is there a canonical list of publicly available data? Are there meta-indexes to indexes of the stuff? How does one go about keeping up to date with all of the numbers that are surely being published every day?

Can I specify the order in an ORDER BY clause in a SQL Statement?

xetere — Thu, 19 Feb 2009 09:18:27 -0800

Can I specify the order in an ORDER BY clause in a SQL Statement? If not then what is the best way of doing the following query? Can I specify the order in an ORDER BY clause in a SQL Statement so that it is neither ASC or DESC?
or if I can't - which I think is the answer after searching in vain on Google - then what is an alternative to solve this seemingly easy problem.

Employee directory. Each employee is assigned to a department. Many but not all employees are assigned to a second department. The directory lists the employees by last name.

What needs to happen is that if a department (say # 3) is queried, all persons who are assigned either primarily or secondarily to that department get listed, but the order must be by Primary department, lastname, THEN just lastname.

Thus if a query is for department 3 then the output must be

Name        DEPT        SECONDARY DEPT
Johnson, J. 3            1
Jones,M.   3
Moss, P .   3             4
Adams, J. 4              3
Johnson.Q 1             3

(note that it doesn't matter what department the secondary users are in, that can be sorted by just last name)

The select statement I have is something like this;

SELECT employee.id, employee.lname, employee.fname, employee.department, employee.secondarydepartment FROM EMPLOYEES WHERE (employees.department = "3" OR employees.secondarydepartment = "3" ORDER BY *huh*? (this is where I am stuck.)

Problem is there are at least six departments, each assigned a number, so using ORDER BY Department won't work if it is in the middle.

I am toying with the ideal of creating a second table, something like employeesaffiliation and then doing a JOIN, complicated, but I think I can do it - unless SQL experts can point me on the right path. I admit I am out of my league with more than simple queries.

Function databases?

archagon — Thu, 29 Jan 2009 19:06:37 -0800

I listened in on an engineering design lecture a few days ago, and was fascinated to find out that there are apparently databases out there that contain all the different ways some common functions can be accomplished - for example, getting liquid out of a cup. What are these databases called and where can I find them?

How to get Filemaker and HomeBase databases on the web?

nessahead — Tue, 28 Oct 2008 14:41:17 -0800

How do I build a website to catalog historic documents, photographs, autographs, books and magazines out of FileMaker and AbeBooks.com HomeBase databases? My uncle collects rare books and magazines, autographs, photographs and other documents and wants to put a catalog up on the web that he can direct people to.

He uses HomeBase from Abebooks.com to catalog the books and magazines. He uses (an old version of) FileMaker to catalog the documents, photograph, autographs, etc. I'm not familiar with either of these applications but told him I'd try to help him figure out how to get it published to his own domain name.

FileMaker seems to support publishing to the web. And perhaps the best solution for the books is to go with Abebooks.com or Alibris.com and just link to those sites from his website?

MySQL joins

dydecker — Thu, 21 Aug 2008 21:58:06 -0800

I have a problem joining three tables in MYSQL. I have three tables I'd like to join and I can't quite get the SQL syntax right. Table 1 has two columns of numbers. I'd
like to join one of these columns with the same column in table 2, and the other column with its match in table 3.
So you could say there are two primary keys perhaps?

Could any SQL expert give me the syntax to return a list of all entries in table 1 with either a match in table 2 or 3?

Obviously I'm a bit of a beginner at this ;)

CSV Help!

archimago — Wed, 20 Aug 2008 11:27:06 -0800

I need to save an Excel csv file of numbers with leading zeroes (e.g., 00987654). Everytime I save the file and reopen it, it drops the leading zeroes. I know how to get them to appear (custom create a number format of 00000000) but they are not saved this way in the csv file. Help! The file has to be a csv file because I am importing data into a database, and those numbers are unique IDs assigned by the database and cannot be changed to ones without the leading zeroes.

Distillation - VLE Database?

whatzit — Mon, 07 Jul 2008 20:09:29 -0800

Separation filter: Where do you get your VLE diagrams, or the boiling point data used to make them? Bonus points if you have a link that will walk me through sizing (trays, diameter, height...) of a distillation column. I have Perry, King, and some other reference books, but no textbooks, and no library to speak of. The system I'm specifically working with is C3H6-C3H8. I hoped I would never need to do this in real life, but I was wrong.

How can I import a 46GB database into MSSQL without ending up with a 46GB transaction log?

odinsdream — Thu, 29 May 2008 12:13:57 -0800

I need to import a database into MS SQL but I do not want to end up with a transaction log as large as the database. What's the best way to do the import? We're working with a software vendor to migrate their database into MS SQL. They set up an ODBC connection to their database and then use Enterprise Manager Data Transformation Services (DTS) packages to copy the tables into an MS SQL database.

However, at the end of their test pass we had a 46GB database file and a 46GB transaction log.

What's the best way to do this without creating such a giant log?

Book catalogue with web OPAC?

Webbreader — Tue, 27 May 2008 03:30:53 -0800

Can anyone recommend a simple open source library management system / book database which offers a searchable web front-end / OPAC and which accepts imported data without MARC records (ie CSV or tab-delimited)? My organisation has a large collection of books which have been catalogued in Readerware, which is a basic book database with an auto-cataloguing function. We would like to make the catalogue available to outside users via a searchable web database. It needs the following features:

* Users can search by different fields (author, collection, keyword, etc).
* Records can be imported in CSV or tab-delimited format (my records do not have MARC data).
* Easy to set up and administer for a non-expert (with some qualified IT expertise on call).

My current server is a Windows server running Microsoft SQL 2005; there is some flexibility in server options, however.

It would be fine for this to be a static database without any significant management options - as long as I can import my existing data and allow other users to see it, my basic needs are met. However, more functionality is obviously better.

In the long term, this collection of books will be properly catalogued and we will want a proper library management system. So, a system with the potential to meet these needs would be nice, but my priority is something quick and easy.

I have some familiarity with Ruby on Rails, so a Rails system might be easier for me to administer, but this is not essential.

I've looked at some open source LMS such as Koha, and they may meet my needs, but I'm definitely in need of some input from the hivemind!

Synchronize databases....3...2...1... NOW!

advicepig — Thu, 03 Apr 2008 18:50:53 -0800

What's the best way to keep data one way synchronized between an internal Mircosoft SQL Server database and a web facing mySQL database? So we have an internal system that's based on SQL Server. I've created numerous DTS packages for moving SQL Server data from the inside SQL Server to a SQL Server at our ISP with some complex transformations, but what I really need to do now is move data from our internal SQL Server to a mySQL database.

Here are some requirements and complications.

1. The internal SQL Server system doesn't have any obvious way to see what's changed in a days work.

2. The mySQL server should know when a row was last modified!

3. We can have as many supplemental tables on either database, but no changes can be made to the exisiting SQL Server tables.

4. Assume SQL Server 2000 and the mySQL version is rather up to date, but completely flexible on the mySQL side.

Are any of those database synchronization tools out there decent? The internal database is mission critical, so it's vital that it not be fouled up.

Replication in MySQL 5.0

kaozity — Thu, 06 Mar 2008 10:16:12 -0800

Does anyone know of a quick and clear guide to set up my MySQL Replication in MySQL 5.0? Does anyone know of a quick and clear guide to set up my MySQL Replication? I want a master and multiple slaves. All machines are on the same LAN. Must be for MySQL 5.0. Of course, I want it to automatically replicate itself.

Database jobs other that D.B.A.

badger11 — Sun, 02 Mar 2008 09:24:37 -0800

Other than being Database Admin, what other kinds of database jobs are there? Looking to leave tech support and maybe server administration I'm tired of being in tech support and server administration. I've always enjoyed designing, building, tinkering, etc databases (granted not at an enterprise level, more of a hobbyist). Mostly with MySQL and some SQL Server 2005. But I'm not sure I want to be a DBA. What other areas can I work with databases in?

It's not you - my CPU says yes, but my database say no

markovich — Fri, 16 Nov 2007 18:54:18 -0800

How many database reads/writes are okay per web page view? Also, what is better, big tables or lots of join tables? I'm starting out in Web based development after years of C++, and this is the one topic I cannot seem to find a lot of information on.

When I have a database in the back end, I will need to read and write data to this database. Each page-open for each user will need to query the database. What is excessive, and what is good? Is 10 individual queries per page open too much? Is 100 too much? If I am only transfering small amounts of data, does it make a difference?

Let's say for example I have a user name, and I need to get some information about him whenever he logs in - say for example, his article view count. This changes every time he opens the page, and needs to be persisted in the database. Do I read and write to the database for every page open?

This is assuming no-caching.

The second part of the question - is it better to have lots of tables storing small amounts of data which are linked to each other using relationship ids, or is it better to use single tables with lots of columns? Which is more efficient?

And finally, in this example, what is better practise:

If I have 10 items in a database, and I need to associate the 10 items with 10 other values from the same database. Is it better to query the 1st item from the db, then query the matching item, and associate them? Or would it be better to query 10 of the first item at once, then 10 of the second item at once, then use a double loop to associate them? The first is heavy on database, the second is heavy on CPU. What is more scalable in the large scale?

name this bug

Tuwa — Sat, 16 Jun 2007 20:09:31 -0800

Why doesn't this code from O'Reilly's PHP and MySQL work for me? pp. 172-173 of the book gives the following code, minus the print "wtf #"; lines, which I added to try to help me figure out where things went wrong.

The results I'm getting--with, naturally, the database name, user account, and password changed to what they should be--are an HTML page with the total content "wtf 0". Or, in other words, all of the page having to do with connecting to the database, querying it, and printing the results, fails.

I've checked the user account name, the password, the database, and the table several times. Convinced I've missed something, I've then typed each field into another page to paste from there into both the account management screens and into the PHP, ensuring that they all match. It's still not working.

I am not getting any error messages, just that puzzling single line of output followed by nothing more.

I've tried flushing the browser cache and even testing the page in different browsers. They all show the same result.

I can't help feeling that this is probably a palm-forehead error, the kind I specialize in, but I don't know how to go about searching for likely culprits when the results are a complete failure to return any results (an error message would be a big help).

Any ideas what I've most likely done wrong?

Apache 2.2.4, PHP 5.2.2, MySQL 5.0

Code below:
<pre>
<?php
print "wtf 0";

// (1) Open the database connection and use the winestore
// database
$connection = mysql_connect("localhost","fred","shhh");

mysql_select_db("winestore", $connection);

print "wtf 1";

// (2) Run the query on the vols through the
// connection
$result = mysql_query ("SELECT * FROM
wine", $connection);

print "wtf 2";

// (3) While there are still rows in the result set,
// fetch the current row into the array $row

print "wtf 3";

while ($row = mysql_fetch_array($result, MYSQL_NUM))
{
// (4) Print out each element in $row, that is,
// print the values of the attributes
foreach ($row as $attribute)
print "{$attribute} ";

print "wtf 4";

// Print a carriage return to neaten the output
print "\n";
}

// (5) Close the database connection
mysql_close($connection);

print "wtf 5";

?>
</pre>

File Dependency System

philosophistry — Tue, 22 May 2007 08:05:02 -0800

Are there file dependency management systems out there? I'm not even sure what this is called. But we have a bunch of assets at our company that are spread over many files, with some assets sharing files. If we want to remove an asset, we have to search all the other assets for dependencies before deleting it.

I'm sure a product like this exists, I don't know what search string to look for, or what concept is out there.

Help a stupid noob find a good MySQL GUI program

Tuwa — Sun, 18 Feb 2007 15:49:28 -0800

Stupid MySQL noob question: is there a free/free (or at least free-as-in-beer) win32 MySQL frontend that will show graphical representations of tables, allow drag-and-drop editing of their relationships, actually import all of my data, and use MySQL vocabulary and syntax for its queries? I have a MS Access database I've been trying to port to MySQL. After a rather fruitless weekend googling and experimenting, I'm tired of trying this on my own and would like advice. I'm sure I'm missing something obvious but I'm still missing it.

For whatever reason, the MySQL migration kit is truncating table data at about the 300th row. So I've switched from using that to the (tedious) process of exporting each table as a .csv through Access, then trying to import each table one by one.

Webyog/SQLyog will not import my data. I've started with the largest table, with the rows titled correctly and in the correct order, with the Excel/.csv-friendly delimiters, and it always returns the message that it either skipped or deleted 2600 rows.

I have found articles talking about using Access as a frontend to a MySQL database. That sounds interesting, but the trouble is that I don't yet have a MySQL database because I haven't been able to import all of the data in any of the tables. A second trouble is that Access uses different SQL syntax than MySQL and I want to use MySQL syntax so I don't have to take on a second hosting plan.

What I want is a GUI/frontend to MySQL which will a) import a .csv file (not a hard requirement, you'd think), b) not cost me any money, c) show a graphical representation of relationships, d) allow the user to edit relationships by dragging and dropping rows in that graphical representation, and e) use MySQL syntax.

I'd keep banging away at this but I need this project off the ground sooner rather than later. And I'm sure I'm missing something obvious but I'm still missing it. Where is this program? I know it exists.

What are the best online resources for scientific data?

jonesor — Thu, 30 Nov 2006 04:40:17 -0800

What are the best online resources for scientific data? I would like to compile a list of online data resources aimed at scientists. I am thinking of things like the Global Population Dynamics Database (GPDD), the HapMap project and TreeBase which are aimed at biologists. However, I would like to compile a list of as many of these resources as possible, right across the sciences.

Ideally these resources would provide free access to data that could potentially provide the basis for completely novel analyses or provide ancillary data that can aid other analyses.

Please can you help me compile the list?

How can a willing to learn non-programmer build something like those user-supported gas price sites?

pjhagop — Wed, 11 Oct 2006 13:18:43 -0800

I'm looking to build a site that allows its users to submit and/or search pricing and availability information for "widgets" at local brick and mortar retail stores. The model I have in mind would be similar to the user-supported gas price sites that popped up about a year ago, in which people entered what they paid for what type of gas, and other users then could type in a zip code and see the best prices near them. So, being a non-programmer (but willing to learn through trial and error), what language/framework/open-source code do I use to build this? Instead of gas, I'm looking to support a broader set of products. OK, I'm talking about beer.

I want to allow a user to enter their zip and get a list of the stores near them, the types of microbrew beer they sell and at what price. Or they could enter the name of the beer they seek, and up would pop a list of the stores that have it and what they're selling it for.

At the same time, they should be able to enter what they paid for beer and where they bought it. I'll need to allow users to enter new types of beer and new locations.

I'd love to also have a slick google maps interface, but that's really phase 2 type stuff.

I want to use this as an opportunity to teach myself something about programming and databases, but I'd rather start with an existing code base that will let me crack it open and see how it ticks. I tend to learn technical things more easily that way.

I'm doing this as a hobby, and want to do it on the cheap. Y'know, free. As in beer.

So, thoughts? Advice? Warnings?

Looking up name variations

zixyer — Thu, 28 Sep 2006 08:33:12 -0800

I find myself often having to look up several lists of names in a database. I would like to automate the process, but the problem is I'm usually given a nickname or variant on a person's full name (i.e. I'm given "Jim" and the name is "James" in the database). I think the ideal solution would be if I had some sort of list of names with all common variants for each. Then, when I'm automating the lookup, I could look up every possible permutation of each person's name. Does anyone know of such a list or any other common approaches to solving this problem?

Web 2.0 catalogue database collaberative tool?

soulbarn — Fri, 04 Aug 2006 15:57:12 -0800

A Web 2.0 way to collaberate with one other person on creating a bibliography/catalogue of references.... I normally use a spreadsheet for this, but the person who is helping me won't always have access to her computer or be able to load MS Excel from a remote computer. We are working to enter about 500 items in a bibliography; the info need to be sortable, both alphabetically and for some numeric values. There will probably be up to ten fields per item. I know there are some collaberative spreadsheets, like Google Spreadsheet, and that is a pretty good option for the basics. However, I would like to turn this into more than a list, and adding these features would be great: the ability to also store URLS that we've used as sources, as well as add new ones as we work. the ability to tag (and search by tags) the items we enter. even the ability to store images and sound files as entries... Does anyone know of any online tools that can do this? Or have any other suggestions?