International characters are garbled at Twitter

Hi Peter,
I just tried it again and got the same problem.
http://twitter.com/pixietest/status/1...

Cool this is fixed now. :)

Sorry about that meant to drop a reply this morning that we fixed it last night. But yea its fixed. :D

Let us know if ya need anything,
~Peter

Now here's another thing:

http://pikchur.com/0w3
http://twitter.com/a_twitter/status/1...

Compare text in those URLs. As you can see, on Twitter, the message text was truncated at a wrong position - in the middle of an atomic character somewhere between 18th and 19th character, which not only makes the last character broken, but makes Pikchur pretty much unusable with CJKV because you can't say much in 18 characters.

I guess that's because you count characters based on bytes? FYI, one Japanese character is typically comprised of 3 bytes in UTF-8, and for what it's worth, if you use some sort of JSON library, it's encoded like "\u3042" which goes up to 6 bytes per character.

On Ruby 1.8.x, use of $KCODE and split(//) should do the trick.

$ irb
>> $KCODE="UTF8"
=> "UTF8"
>> "ああああああああああ".length
=> 30
>> "ああああああああああ".split(//).length
=> 10
>> "ああああああああああ".split(//)
=> ["あ", "あ", "あ", "あ", "あ", "あ", "あ", "あ", "あ", "あ"]
>> "ああああああああああ".split(//)[0..4].join
=> "あああああ"

If you run MySQL, it's notorious for the complicated setting with the default of latin-1 and to make everything work under UTF-8 you need 2 lines under [mysqld] section in my.cnf.

[mysqld]
default-character-SET = utf8
skip-character-set-client-handshake

See:
http://word.wardosworld.com/?p=164

Hope this helps.

cool, thanks for the trick. :D

Will try and get that UTF-8 char count issue fixed asap.

I'll let ya know when its up. :)

Thanks again.
~Peter