Comments on: How do you deal with Chinese characters that can't be represented in 16 bits?

Question: How do you deal with Chinese characters that can't be represented in 16 bits?

bonecrusher — Fri, 04 Apr 2008 08:48:52 -0800

How are people dealing with >16 bit Unicode code points? Specifically, in languages like Java, C# and C++, which assume 16 bit characters (I believe), how are you supporting GB 18030? I would suspect that the various languages' methods like substring(), charAt(), operator[], etc can't be safely used in China. If your wstring, say, contains a Chinese string, then .size() doesn't tell you how many characters are in it, right? On a related note, what interesting Chinese characters require more than >16 bits? I'm thinking about making a short presentation for my co-workers on this subject and I'd like to have some interesting examples. (Oh, and I'm going to run any examples by my Chinese colleagues first, so don't bother trying to make me say "penis" or something in front of my co-workers :-))

By: tachikaze

tachikaze — Fri, 04 Apr 2008 09:05:41 -0800

The unicode term you're looking for is "surrogate pair" or "surrogate code point".

By: bonecrusher

bonecrusher — Fri, 04 Apr 2008 09:15:23 -0800

Right, but if there are surrogate pairs in your wstring, and you choose an unfortunate length for your substr, you're in trouble, aren't you?

By: hattifattener

hattifattener — Fri, 04 Apr 2008 09:33:45 -0800

Yes, but you already have that problem, since Unicode includes combining marks. If you subdivide a string after the base mark and before a combining mark you'll get unexpected results too.

The OpenStep string classes have some methods for "give me the range of code points that make up the complete character at this point in the string", which should work for surrogate pairs as well as combining marks. Dunno if Java has an equivalent.

By: hattifattener

hattifattener — Fri, 04 Apr 2008 09:53:40 -0800

(And for "complete character" I probably should write "grapheme").

By: hattifattener

hattifattener — Fri, 04 Apr 2008 09:56:57 -0800

(no, wait, "grapheme" is a slightl yhigher level concept than I'm looking for. Ehhh, text is hard.)

By: burnmp3s

burnmp3s — Fri, 04 Apr 2008 09:58:29 -0800

In most languages you can avoid these problems by using regular expressions to do string manipulation instead of methods that involve substring() and charAt(). As long as the language's regex engine supports Unicode, it will usually have a way to select individual gramphemes regardless of the byte representation.

This article has a good overview of using regexes with Unicode and the various issues to watch out for.