# What the � **URL:** \ **Category:** Factual Questions\ **Created:** [April 24, 2013, 8:25pm UTC](https://boards.straightdope.com/t/what-the-a/656619 "2013-04-24T20:25:49Z")\ **Posts on this page:** 10\ **Page:** 2
**Author:** ![TimeWinder](https://avatars.discourse-cdn.com/v4/letter/t/bcef8e/32.png) [@TimeWinder](https://boards.straightdope.com/u/TimeWinder)\ **Post date:** [April 25, 2013, 5:03pm UTC](https://boards.straightdope.com/t/what-the-a/656619/21 "2013-04-25T17:03:27Z")
> [@scr4](#): > > Yes, they’ve been working on the problem for over 2 decades. Yes, but that’s not what the quote I was responding to said. > [@scr4](#): > > And your company spends “billion or so dollars a year” dealing with character set compatibility issues?? Even if you work for Google or Apple, I doubt they spend 2% of the revenue on this issue alone. Unicode support doesn’t come for free, has to be implemented across a wide range of products, and has to deal with third-party systems that include non-standard encodings, don’t support both byte orderings or all the UTF word sizes, with a variety of line endings (both standard and not)–and that’s not even counting that many of these systems don’t handle surrogates and think it’s “OK.” Or the “Unicode” fonts that support only odd subsets but are in standard use anyway and need to have characters substituted from other ones – again, across hundreds of products. I’ll admit I have no cite for it, but I’d be stunned if my billion dollar estimate wasn’t low – this is the foundation on which all localization is made possible; character encoding issues are _at least_ 2% of my work, and I’m not in a particularly text-oriented part of the company. We’ve got libraries to standardize this stuff, of course, but no company is an island (especially companies that make web browsers), and we have to work and play well with others, including the vast majority of software developers who don’t have the resources to even understand all the nuances of unicode, much less the time and dollars to implement them. The unicode base specification alone is over six hundred pages long, and that’s not counting the almost four thousand pages of amendments, proposed changes, exceptions, and committe recommendations that have been adopted by various organizations. ---
**Author:** ![Dr.Strangelove](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dr.strangelove/32/6613_2.png) [@Dr.Strangelove](https://boards.straightdope.com/u/Dr.Strangelove)\ **Post date:** [April 26, 2013, 1:58am UTC](https://boards.straightdope.com/t/what-the-a/656619/22 "2013-04-26T01:58:42Z")
> [@scr4](#): > > Even if you work for Google or Apple, I doubt they spend 2% of the revenue on this issue alone. TimeWinder’s location says “Redmond”, so it’s not hard to guess the multi-billion software company he works for. Given their position in the market and the effort they spend on compatibility, a billion a year probably \*is \*about right. And Google or Apple probably don’t spend that much, because their product lines are more focused and somewhat more isolated. ---
**Author:** ![BigT](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/bigt/32/12044_2.png) [@BigT](https://boards.straightdope.com/u/BigT)\ **Post date:** [April 28, 2013, 6:59pm UTC](https://boards.straightdope.com/t/what-the-a/656619/23 "2013-04-28T18:59:18Z")
You don’t need to fix the program, just run it through a charset converter. ---
**Author:** ![Telemark](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/telemark/32/372_2.png) [@Telemark](https://boards.straightdope.com/u/Telemark)\ **Post date:** [April 28, 2013, 7:40pm UTC](https://boards.straightdope.com/t/what-the-a/656619/24 "2013-04-28T19:40:40Z")
> [@BigT](#): > > You don’t need to fix the program, just run it through a charset converter. But if those characters can’t be represented in the program in question then what do you do with them? Lots of programs don’t support multi-byte characters to begin with. ---
**Author:** ![elmwood](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/elmwood/32/3060_2.png) [@elmwood](https://boards.straightdope.com/u/elmwood)\ **Post date:** [April 28, 2013, 7:59pm UTC](https://boards.straightdope.com/t/what-the-a/656619/25 "2013-04-28T19:59:29Z")
> [@Little\_Brother\_1](#): > > Why don’t computers automatically “translate” text so that punctuation and other characters in web pages and e-mails aren’t replaced by weird symbols, e.g. “— In my experience, I found it’s not so much a Unicode problem as it is a Microsoft Word problem. MS Word has an autocorrect feature that, by default, will change straight quotation marks to “curly” or “smart” quotes, three periods to an ellipsis, and so on. If a document autocorrected in such a manner is copied and pasted into a plain text format, and in turn displayed on the Web, the result is often strange symbols (to our English-reading eyes, anyhow). A hypothetical example: the [code point](http://en.wikipedia.org/wiki/Unicode#Code_point_planes_and_blocks) of a smart right quote in the Arial font in Word 2003 might be the same as for the code point of an inverted Spanish question mark in Unicode. What looks like ❝Hello world❞ in MS Word will display as ¿Hello world¥, or something similar. Ultimately, it comes down to “garbage in, garbage out.” How would a program know you wanted a right curly quotation mark, when you really wanted a Spanish question mark, or vice versa? ---
**Author:** ![Derleth](https://avatars.discourse-cdn.com/v4/letter/d/b9e5f3/32.png) [@Derleth](https://boards.straightdope.com/u/Derleth)\ **Post date:** [April 28, 2013, 10:43pm UTC](https://boards.straightdope.com/t/what-the-a/656619/26 "2013-04-28T22:43:33Z")
> [@Telemark](#): > > But if those characters can’t be represented in the program in question then what do you do with them? Lots of programs don’t support multi-byte characters to begin with. Heh. First, one character set’s multi-byte (or nonexistent) character is another character set’s single-byte character. Thinking about text in those terms is broken anyway; characters are characters, not bytes or byte sequences, and the complexities around actually handling them in an intelligent way are sufficient to justify leaning on special-purpose libraries as often as possible. Your question is a prime example of the complexities here: In some character sets, like the Latin character set used in Western Europe, there are simple rules to take you from one country’s national variant to another’s in a fashion everyone involved agrees is ‘lossless’. For example, the German ‘ö’ is universally agreed to map to ‘oe’ whenever the ‘ö’ is unavailable. Similar techniques apply generally, reducing the problem to a lookup table. Moving Eastwards, Cyrillic and Greek text can _almost_ be mapped to the Latin alphabet using lookup tables as well, but the ‘almost’ will make you look like a fool if you don’t have some extra intelligence to go along with it. Moving Southwards, you run into the Semitic written languages (Arabic, Hebrew, Syriac, etc.) and the lookup table scheme breaks down entirely if you expect anything intelligible to non-experts to come out the other end. The big problem is partially the simple fact Semitic scripts are abjads, where vowels are very often not written but can be inferred from context. The main problem, though, is that transliteration is a social convention, and the social conventions surrounding transliteration out of Semitic abjads are nothing if not hairy and inconsistent, as the Nine Billion Names of Gaddafi will attest. Going into the Far East, trying to do anything with writing systems based on Han characters (Traditional Chinese, Simplified Chinese, Kanji, Hanja, etc.) with a lookup table is idiocy and will only result in utter nonsense. It’s likely you’ll need a human to do anything with those writing systems. So that’s one tiny example of why Unicode is as complex as it is compared to everything that came before: It’s the only text encoding standard that even begins to be complex enough to grapple with the world’s writing systems. ---
**Author:** ![eschereal](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/eschereal/32/18939_2.png) [@eschereal](https://boards.straightdope.com/u/eschereal)\ **Post date:** [April 28, 2013, 11:38pm UTC](https://boards.straightdope.com/t/what-the-a/656619/27 "2013-04-28T23:38:13Z")
> [@elmwood](#): > > A hypothetical example: the [code point](http://en.wikipedia.org/wiki/Unicode#Code_point_planes_and_blocks) of a smart right quote in the Arial font in Word 2003 might be the same as for the code point of an inverted Spanish question mark in Unicode. What looks like ❝Hello world❞ in MS Word will display as ¿Hello world¥, or something similar. I do hope you are not implying that the Word .doc format encodes text based directly on font glyph mapping tables. I always though MS was a bit lazy and slipshod in their software designs, but if that were the case, the situation would be much worse than I had imagined. ---
**Author:** ![Blakeyrat](https://avatars.discourse-cdn.com/v4/letter/b/ecd19e/32.png) [@Blakeyrat](https://boards.straightdope.com/u/Blakeyrat)\ **Post date:** [April 29, 2013, 11:51pm UTC](https://boards.straightdope.com/t/what-the-a/656619/28 "2013-04-29T23:51:38Z")
> [@TimeWinder](#): > > A system badly enough designed that it is even possible to be used incorrectly by the majority of people isn’t a people problem, it’s a computer problem. > > However, I’m glad that this was perfected for “decades” (HTML was created in 1990, so they must have been quick about it) – that means that the billion or so dollars a year my company spends on it can now be reclaimed so that we can blow it all on hats. Sorry to step on toes (or whatever, you sound angry.) But I stand by what I said: this is a solved problem, it was solved in HTML4 at the absolute latest (release date: 1997, decades ago), and if you are still wrangling with it, it’s due to: 1. Your company developing software wrong in the first place, 2. Your company failing to upgrade older software as standards changes, 3. Your company relying on other companies that have one of the above two problems. In other words, it’s a people/management problem, not a technology problem. The technology’s solid. That said, I do completely agree that in an ideal universe, computer technologies would be impossible for people to misuse, and judging by that standard, HTML has been a failure in that respect-- since you can fail to specify a character encoding and it still “mostly” works, when it should probably simply fail and display nothing. But we don’t live in an ideal universe, we live in this one. So. Billions of dollars a year? On character encoding issues? Seriously? ---
**Author:** ![Telemark](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/telemark/32/372_2.png) [@Telemark](https://boards.straightdope.com/u/Telemark)\ **Post date:** [April 30, 2013, 1:12am UTC](https://boards.straightdope.com/t/what-the-a/656619/29 "2013-04-30T01:12:18Z")
> [@Blakeyrat](#): > > Billions of dollars a year? On character encoding issues? Seriously? Internationalization is much more than just character encoding issues. ---
**Author:** ![Blakeyrat](https://avatars.discourse-cdn.com/v4/letter/b/ecd19e/32.png) [@Blakeyrat](https://boards.straightdope.com/u/Blakeyrat)\ **Post date:** [April 30, 2013, 10:50pm UTC](https://boards.straightdope.com/t/what-the-a/656619/30 "2013-04-30T22:50:49Z")
> [@Telemark](#): > > Internationalization is much more than just character encoding issues. Yes. But all we’re talking about in this thread is encoding issues. So the cost of (the rest of) internationalization is irrelevant. Unless TimeWinder was trying to change the topic and I didn’t pick up on it. [Previous page](https://boards.straightdope.com/t/what-the-a/656619.md?page=1)