Comments on: techie SQL dupe problem

Question: techie SQL dupe problem

derbs — Fri, 12 Aug 2005 08:57:26 -0800

Any SQL gurus out there? I'm stuck...

Hello.
i'm trying to do a pretty simple SQL dupe-finding operation. I'm stumped though (MS SQL Server 7 btw)

Say you've got a simple table:

messageid | message | user

and the messageid column is a unique identifier datatype, one of those GUID things (eg. {3803F70B-4C3E-4C69-8C7F-EC534BBC1A8D}), not just an auto-increment.

How can I find dupes when user and message are the same? It's easy when there's an integer-autoincrement to MAX(), but I can't work it out for a GUID...

Any help would be very much appreciated!

By: milkrate

milkrate — Fri, 12 Aug 2005 09:04:53 -0800

SELECT message, user, count(messageid) AS count
FROM table
GROUP BY message, user;

By: McGuillicuddy

McGuillicuddy — Fri, 12 Aug 2005 09:04:57 -0800

select message, user
from simple_table
group by message, user
having count(*) > 1

By: alkupe

alkupe — Fri, 12 Aug 2005 09:06:18 -0800

select user, message, count(*) 'count' from TABLENAME
group by user, message
order by 'count' desc

this third column 'count' will tell you how many instances there are of each combination.

By: derbs

derbs — Fri, 12 Aug 2005 09:38:01 -0800

ah thanks for all your quick replies! But i really need a unique indentifier in the results so i can then go back and delete the other duplicates!

By: uncleozzy

uncleozzy — Fri, 12 Aug 2005 09:44:38 -0800

How about:

select t1.messageid
from table t1 join table t2
on (t1.user = t2.user and t1.message = t2.message)
where t1.messageid <> t2.messageid

By: vacapinta

vacapinta — Fri, 12 Aug 2005 09:49:48 -0800

delete from tablename a
where a.messageid>
(select min(messageid) from tablename b where a.user=b.user and a.messageid=b.messageid)

will delete all duplicate rows.

By: bemis

bemis — Fri, 12 Aug 2005 09:51:11 -0800

Or, perhaps

create table b as (
select min(message_id) as message_id, message, user
from t
group by message, user
)

Depending on the size of the table and the use of the database, this may just be easier; create a new table of just the non-duplicate info, and then drop the old table and rename. It assumes, arbitrarily, that the message_id you want to save is the min one in each case of duplication.

By: derbs

derbs — Fri, 12 Aug 2005 10:35:24 -0800

you know this t1,t2 and tablename a,tablename b...

are these tables you have created on the fly?

By: vacapinta

vacapinta — Fri, 12 Aug 2005 10:37:25 -0800

oops. typo. corrected below.

delete from tablename a
where a.messageid>
(select min(messageid) from tablename b where a.user=b.user and a.message=b.message);

among duplicate rows, this will delete all of them except the one with the lowest messageid. You could also use max or any clause which guarantees a unique row among many rows (I cant think of any other clauses though)

By: vacapinta

vacapinta — Fri, 12 Aug 2005 10:39:36 -0800

are these tables you have created on the fly?

No. they are alternate names for the same table. Many of these SQL statements are joining or nesting the table with itself - thus the need to give it aliases.

By: blue mustard

blue mustard — Fri, 12 Aug 2005 12:12:10 -0800

This deletes all duplicate messages (except the one with the lowest messageid). It assumes your table is called "messages".

DELETE FROM messages WHERE messageid NOT IN
(
SELECT MIN(messageid)
FROM messages
GROUP BY message, user
)

You may want to back up your db before trying this or any other suggestion.

By: derbs

derbs — Fri, 12 Aug 2005 13:41:12 -0800

Thanks a lot everyone - finally got it sorted! All of your answers were great, but vacapinta's was the one that was most useful (although it needed a slight amount of tweaking)

However, most of you were assuming that i had a column with a unique identifier which was a integer, which i did not have originally (the primary key was a guid). But it turns out i needed to have an unique integer as well anyway (for a separate issue) so it's all turned out fine!

thanks again!

By: sad_otter

sad_otter — Fri, 12 Aug 2005 17:22:56 -0800

Also, you could add a compound uniqueness constraint to prevent the duplicate data getting into the DB to begin with.