Comments on: Fuzzy logic search to find international characters.

Question: Fuzzy logic search to find international characters.

Pericles — Thu, 13 Apr 2006 02:57:37 -0800

I've got a database of names, many of which have international characters (e-acute, c-cedilla, o-umlaut, etc). I want the search routine to be clever enough that if I search for "Celik" it'll find c-cedilla-elik, even though "c" and "c-cedilla" are entirely different.

Does a look-up table exist that matches whole range of such non-English letters with their nearest-looking English equivalents? Or can anyone here help me construct one?

I'm thinking o and u umluat, c and s cedilla, o circumflex, Turkish g and undotted-i, Scandinavian o with a line through it, Spanish n, e with a grave and acute, accented a, the dipthongs.

Any more for any more?

By: grouse

grouse — Thu, 13 Apr 2006 03:07:54 -0800

Perhaps someone will have a better or pre-made solution. But one thing you could do is use the Unicode Character Database, which will allow you to decompose, e.g. U+00C7 LATIN CAPITAL LETTER C WITH CEDILLA into U+0043 LATIN CAPITAL LETTER C, U+0327 COMBINING CEDILLA.

By: Goofyy

Goofyy — Thu, 13 Apr 2006 04:11:50 -0800

Don't forget a-umluat.

By: Sharcho

Sharcho — Thu, 13 Apr 2006 04:17:02 -0800

At least for MySQL, you can simply use the LIKE operator,
e.g. SELECT 'áçñ' LIKE 'acn' returns 1, but SELECT 'å' LIKE 'a' returns 0, so it doesn't work in all cases, but it might be good enough for you.

By: public

public — Thu, 13 Apr 2006 04:39:02 -0800

Are you using MySQL?

Switch to a sensible DB, e.g. PostgreSQL.

or

Convert your (VAR)CHAR/TEXT columns to BLOB. I expect it won't try and be clever about things then.

By: grouse

grouse — Thu, 13 Apr 2006 05:01:00 -0800

Convert your (VAR)CHAR/TEXT columns to BLOB. I expect it won't try and be clever about things then.

I don't understand this response. The OP wants more cleverness, not less.

By: Pericles

Pericles — Thu, 13 Apr 2006 05:07:25 -0800

It's an Oracle database.

By: Yeomans

Yeomans — Thu, 13 Apr 2006 05:19:02 -0800

Try DevonThink Pro. I don't know how extensive the "character equivocation" (my term) can get, but it has a stunningly-intelligent fuzzy logic search and can, from my experience, ignore/'equivocate' umlauts.

By: tperrigo

tperrigo — Thu, 13 Apr 2006 06:10:17 -0800

You might want to check out the levenshtein distance algorithm. It calculates the "distance" between two strings (the number of edits it would take to transform 1 string into another). In your case, although "c" and "c-cedilla" are different characters, the levenshtein distance between them would be 1 (so you would find it if you searched for all matches within a certain range). Here is an implementation in Oracle's PL/SQL.

By: nebulawindphone

nebulawindphone — Thu, 13 Apr 2006 06:15:04 -0800

('Course, the levenshtein distance between "C" and "Q" or "j" or "!" is also 1. Pericles wants "Celik" to match "Çelik" but not "Qelik" or "jelik" or "!elik".)

By: Pericles

Pericles — Thu, 13 Apr 2006 06:34:45 -0800

I'm pretty sure a dumb look-up table would do it. User types in Celik, search routine goes to table and sees ç next to "c" and so also searches for that.

By: zpousman

zpousman — Thu, 13 Apr 2006 06:42:46 -0800

To tag along, I have this problem in my iTunes library. Any ideas for solving that problem (other than chaning the names of tons of bands and songs, of course)?

By: adamrice

adamrice — Thu, 13 Apr 2006 07:15:07 -0800

Depending on how you're setting things up, one way to do this is to store a hidden field that has the ASCII-fied version of your accented strings, and whenever someone does a search, run their query string through the same ASCII-fier before doing the search.

Obviously this would get clunky if you're dealing with long blocks of text, but I did use something like this technique when there were only a few searchable fields. The overhead was low and it worked well.

I did a little googling and ran across the following links that may be of use to you:
1. Linguistic sorting & string searching. This is specifically about Oracle.
2. Multilingual databasing techniques. This discusses my ASCII-fied technique, along with a clever variation on it.

By: Pericles

Pericles — Thu, 13 Apr 2006 08:12:16 -0800

Thanks Adam Rice. I did do a little googling (honest!) but wasn't clever enough on my search terms, evidently ...

By: costas

costas — Thu, 13 Apr 2006 08:52:55 -0800

This may be overkill for what you're looking for, but IBM's ICU library (in Java and C++), has transliteration abilities of the type you're looking for. I haven't used it myself (yet anyway), but from the examples they have online for my own target language (Greek), the features are pretty impressive.

By: odinsdream

odinsdream — Thu, 13 Apr 2006 10:15:38 -0800

If you do run a search against an ASCII-only version, be sure you aren't making it more difficult for users who already know how to type international characters to find exactly what they're looking for.

That is, if you search for Jacob, and find Jacöb, great, but if you search for Jacöb and get no results, that's bad.

By: mrg

mrg — Thu, 13 Apr 2006 11:15:19 -0800

have you thought about using soundex or metaphone algorithms for this? they generate a key for the string based on how the string sounds. (both those algorithms were originally intended for use with English but I imagine you could or someone has adapted them for other languages and such.)