Wanted: "Programmatic Handwriting Recognition for Dummies".

April 13, 2010 7:57 AM

How can I learn how to programmatically recognized Japanese handwriting?

Roughly one year ago, I asked a question about what Kanji-learning apps are available to Japanese learners. After reading the answers to that question, and a year of continuous searching and waiting, I've determined that there just isn't an application that perfectly meets my needs.

So, I'm interested in trying my hand at writing it myself. The problem is - how?

I'm already a professional computer programmer, and I don't doubt I could learn how to write iPhone or Android apps in a short period of time. There are plenty of good resources out there for learners of those platforms.

What I don't know how to learn is handwriting recognition. I would like to write something similar to the handwriting input system of Nazotte Oboeru Otona no Kanji Renshu DS, or even Skritter. But I don't really know where to go to learn handwriting recognition algorithims. There must be some way to learn these things, right? I mean the technology's in Microsoft's IME, in lots of Japanese learning apps, in ShinKanji for the iPhone... where are people learning how to do this?

Secondly, even if I knew the algoritihim, I'd still need a database of Kanji to match the results against. Is there a public or creative-commons-type database of Kanji that includes stroke order and shape info? Anyone know where to find it?

tl;dr filter:

1. How can I learn to write a program that recognizes Japanese handwriting?
2. Where can I find a database of Kanji to match handwriting results against?

Thanks MeFites!

posted by Vorteks to computers & internet (9 answers total)

Obligitatory noticed-the-flaw-immediately-after-clicking-post-button correction:

programmatically recognized = programmatically recognize
posted by Vorteks at 7:58 AM on April 13, 2010

As you probably know, kanji are especially well-suited to this problem because of the limited number of strokes and rigorous stroke order.

There's a Mac app called JEDict that is an interface to EDICT and KANJIDIC dictionaries, and it will show the series of strokes used for a character (it can also animate the process of drawing the character). Apparently the author got that stroke information here.
posted by adamrice at 8:17 AM on April 13, 2010

Might be worth getting in touch with this guy, who wrote the linked paper on using neural networks in handwriting recognition for his Master's in Comp Sci at Urbana-Champaign. Looks like he didn't write an algorithm, but he's got some recommended reading that could form a good basis. You might find some helpful folk in the Pleco forums - Pleco Dict is a great Chinese learning tool which utilizes handwriting recognition for Chinese characters. I realize you want to do this for Japanese kanji, but maybe there are some similarities. This article also looks pretty interesting. Here's a database that looks promising.

I'll be interested to hear about your progress. Very interesting project!
posted by xiaolongbao at 8:23 AM on April 13, 2010

I wanted to write a Hiragana/Katakana recognition iPhone app -- but never did. I asked the same questions. So for what it's worth, here were my thoughts:

I figured that the problem was much simpler than OCR, since you knew the actual path that was used to draw the character. I figured with position and angle difference between points along the path, you could come up with a template use that to recognize a drawn character.

I'm sure you could find some basic character templates on the web, but I was worried about copyright issues and also that they would look ugly. I thought it would be a good idea to ask a native-born speaker/artist to draw them. This would be my template: the images shown to the user and what they were trying to match against.
posted by whiskeyspider at 8:33 AM on April 13, 2010

You're looking for pattern recognition; "hidden markov model" will be a useful google term. A coworker of mine opensourced a C++ pattern recognition library called AMELiA. I think it might actually have a handwriting example in the download. There're also a bunch of Openframeworks examples in there, and another friend of mine has successfully used the library in an Objective-C Cocoa app. I suspect you could find similar libraries for most other languages -- JAHMM comes up for Java in a google search.
posted by Alterscape at 9:52 AM on April 13, 2010 [1 favorite]

I'm already a professional computer programmer, and I don't doubt I could learn how to write iPhone or Android apps in a short period of time. There are plenty of good resources out there for learners of those platforms.

The hard part will be the recognition.

1. How can I learn to write a program that recognizes Japanese handwriting?
2. Where can I find a database of Kanji to match handwriting results against?

You'll need a lot more the then a database, you need tons of writing samples to learn against. Character recognition was big in AI and machine learning years ago. But learning Kanji? It could be tricky, especially with calligraphic characters.

What I would do in your position is use Google Scholar and search for papers on Kanji Character Recognition. this article looks promising, but it seems to be behind a paywall.

And don't forget to search for Hanzi Character Recognition. Doing that brought up this book which has Chapter on processing radicals

But actually the best search term is probably Chinese character recognition brings up 391k results, compared to 12k for kanji and 541 for Hanzi. This article is free to download, but was written in 1976. And This book on amazon on

"Arabic and Chinese Handwriting Recognition" Arabic and Chinese Handwriting Recognition: Summit, SACH 2006, College Park, MD, USA, September 27-28, 2006, Selected Papers (Lecture Notes in Computer ... Vision, Pattern Recognition, and Graphics) (Paperback)

Then there's this one:Optical Chinese character recognition using probabilistic neural networks (PDF). Neural Networks were really effective for learning Latin characters, and probably other small alphabets.

Oh, and "Chinese Character OCR" also seems to get good results, many of which are downloadable, such as High Performance Chinese OCR based on Gabor features, Discriminative Feature Extraction and Model training and Online recognition of Chinese characters: the state-of-the-art

Oh, and when you read papers like this, look at the citations, not just the articles themselves. Google scholar is a great way to dig through and find other papers, for example the "cited by" will find stuff that's usually pretty similar.

As far as datasets go the HIT-MW dataset looks promising. It's designed for Chinese, and I'm not seeing any Japanese specific ones, although I'm sure there are some out there if you look.

But the thing to understand is that this stuff isn't easy, it's significantly more complicated then "regular" programming. You'll probably need to have a strong understanding of some pretty complicated math.

---
Now, if you just want to do something where you show the user a character, and they have to trace it exactly, that might be a little easier.
posted by delmoi at 10:59 AM on April 13, 2010 [1 favorite]

1. How can I learn to write a program that recognizes Japanese handwriting?

Dunno. I downloaded this recently, and have yet to determine how well it works...but there ya go. Also, (I don't particularly care for this app but): another related android app. Sorry for the spammy links, I don't know if there's a "good" Android market search web site.

Also, I've played with Tagaini and I like it a lot. Got stroke order animations in there, and is open source.

Sorry I can't give you anything more concrete than that, others have better info regarding algorithms and such.

2. Where can I find a database of Kanji to match handwriting results against?

This seems like the most obvious place to start.
posted by dubitable at 12:23 PM on April 13, 2010

Thanks for the answers. At least I have a better idea of where to look for more information.

I don't think I need to do anything as complicated as OCR, since I won't be recognizing images, raster data, or anything like that. The user input will be in vector format, and will include stroke number and direction, which would make things easier.

Also, although being able to recognize handwriting well enough to find any character the user enters (say, for dictionary lookup) would be ideal, since I'm thinking of doing more of a flash-card type program I realized after thinking about it some more that my program really only needs to distinguish between 3 possible inputs:

1. Success - You drew the expected character correctly
2. Near Miss - You drew the expected character, but with the wrong stroke order, and should be given a refresher on how to draw this character
3. Failure - You drew some other character which was not the correct answer.

That should make the problem easier, right?
posted by Vorteks at 6:29 AM on April 14, 2010

Yes, it's not OCR. I never implemented this, but I was thinking you could do it by measuring the angles between the points in each line. Set a threshold angle and only pay attention to those that surpass it. Come up with a way to identify a line shape based on the most salient angles. You would also have to measure direction (i.e., it could be the correct line shape, but in the wrong direction).

You see this kind of line shape recognition stuff in Nintendo DS games all the time. I doubt they are doing "character recognition using probabilistic neural networks" or whatever.
posted by whiskeyspider at 4:03 PM on April 14, 2010