Mercurial > sdl-ios-xcode
annotate test/utf8.txt @ 4384:6800e2560310 SDL-1.2
Fixed bugs #882 and 865, re-opening bug #634
Ronald Lamprecht to SDL
Hi,
Sam Lantinga wrote:
The problem with that fix is that it breaks IME events again. Maybe
we can handle keyboard events differently to prevent this issue?
Spending an hour reading MSDN, analysing SDL and another hour testing the reality on XP I am really wondering how patch r4990 could have ever worked in any situation. It's main effect is to break the unicode translation and causing spurious activation events!
Why does TranslateMessage(&msg) nothing useful? Simply because it does not affect "msg" at all! All keyboard events are dispatched without the slightest change (see MSDN). TranslateMessage() just appends additional WM_CHAR, WM_DEADCHAR, WM_SYSCHAR, WM_SYSDEADCHAR event messages to the queue. But I could not find any SDL event handling routine that catches these events and transforms them to proper SDL keyevents while eliminating the corresponding WM_KEYDOWN, etc. events. Thus any IME input like the '@' generated by "Alt + 6(Numpad) + 4(Numpad)" is simply lost.
But the situation is even worse! Up to r4990 the TranslateKey()/ToUnicode() calls did evaluate dead keys and did deliver proper key events for subsequent key strokes like '´' + 'e' resulting in 'é'. ToUnicode() needs proper key state informations to be able to handle these substitutions. But unfortunatly TranslateMessage() needs the same state information and eats it up while generating the WM_CHAR messages :-( Thus the current 1.2.14 breakes the partial IME support of previous releases, too.
The key state race condition between ToUnicode() and TranslateMessage() requires to avoid any ToUnicode() usage for receiving proper WM_CHAR, etc. messages generated by TranslateMessage(). (Yes - the '@' and 'é' appear as WM_CHAR messages when unicode is switched off).
The spurious SDL activation events are *not* caused by additional WM_ACTIVATE Windows messages! Besides DIB_HandleMessage() SDL_PrivateAppActive() is called by another source which I am not yet aware of - any hints?
Thus I do strongly recommend the deletion of the TranslateMessage(&msg) call as a quick fix.
A proper support of unicode and IME requires a clean SDL keyboard input concept first. Which SDL keyboards events should be transmitted to the app when the user presses '´' + 'e' ? Within the current unicode handling the first key stroke is hidden. Even though ToUnicode() delivers the proper key SDL does ignore it in TranslateKey(). Just the composed key event is transmitted to the app. That is what you expect for text input, but the app can no longer use keys like '^' as a key button because it will never receive a key event for it!
With a given concept it seems to be necessary to regenerate SDL key events out of the WM_CHAR, etc. events and to drop all related direct WM_KEYDOWN, etc. events while the remaining basic WM_KEYDOWN, etc. events would still have to result in SDL key events.
Anyway the source of the spurious WM_ACTIVATE should be located to avoid future trouble.
Greets,
Ronald
author | Sam Lantinga <slouken@libsdl.org> |
---|---|
date | Tue, 17 Nov 2009 04:59:13 +0000 |
parents | 4d711949cd9a |
children |
rev | line source |
---|---|
1501
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
1 UTF-8 decoder capability and stress test |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
2 ---------------------------------------- |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
3 |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
4 Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2003-02-19 |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
5 |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
6 This test file can help you examine, how your UTF-8 decoder handles |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
7 various types of correct, malformed, or otherwise interesting UTF-8 |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
8 sequences. This file is not meant to be a conformance test. It does |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
9 not prescribes any particular outcome and therefore there is no way to |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
10 "pass" or "fail" this test file, even though the texts suggests a |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
11 preferable decoder behaviour at some places. The aim is instead to |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
12 help you think about and test the behaviour of your UTF-8 on a |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
13 systematic collection of unusual inputs. Experience so far suggests |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
14 that most first-time authors of UTF-8 decoders find at least one |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
15 serious problem in their decoder by using this file. |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
16 |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
17 The test lines below cover boundary conditions, malformed UTF-8 |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
18 sequences as well as correctly encoded UTF-8 sequences of Unicode code |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
19 points that should never occur in a correct UTF-8 file. |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
20 |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
21 According to ISO 10646-1:2000, sections D.7 and 2.3c, a device |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
22 receiving UTF-8 shall interpret a "malformed sequence in the same way |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
23 that it interprets a character that is outside the adopted subset" and |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
24 "characters that are not within the adopted subset shall be indicated |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
25 to the user" by a receiving device. A quite commonly used approach in |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
26 UTF-8 decoders is to replace any malformed UTF-8 sequence by a |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
27 replacement character (U+FFFD), which looks a bit like an inverted |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
28 question mark, or a similar symbol. It might be a good idea to |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
29 visually distinguish a malformed UTF-8 sequence from a correctly |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
30 encoded Unicode character that is just not available in the current |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
31 font but otherwise fully legal, even though ISO 10646-1 doesn't |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
32 mandate this. In any case, just ignoring malformed sequences or |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
33 unavailable characters does not conform to ISO 10646, will make |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
34 debugging more difficult, and can lead to user confusion. |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
35 |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
36 Please check, whether a malformed UTF-8 sequence is (1) represented at |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
37 all, (2) represented by exactly one single replacement character (or |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
38 equivalent signal), and (3) the following quotation mark after an |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
39 illegal UTF-8 sequence is correctly displayed, i.e. proper |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
40 resynchronization takes place immageately after any malformed |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
41 sequence. This file says "THE END" in the last line, so if you don't |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
42 see that, your decoder crashed somehow before, which should always be |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
43 cause for concern. |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
44 |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
45 All lines in this file are exactly 79 characters long (plus the line |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
46 feed). In addition, all lines end with "|", except for the two test |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
47 lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
48 U+0000 and U+007F. If you display this file with a fixed-width font, |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
49 these "|" characters should all line up in column 79 (right margin). |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
50 This allows you to test quickly, whether your UTF-8 decoder finds the |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
51 correct number of characters in every line, that is whether each |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
52 malformed sequences is replaced by a single replacement character. |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
53 |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
54 Note that as an alternative to the notion of malformed sequence used |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
55 here, it is also a perfectly acceptable (and in some situations even |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
56 preferable) solution to represent each individual byte of a malformed |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
57 sequence by a replacement character. If you follow this strategy in |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
58 your decoder, then please ignore the "|" column. |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
59 |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
60 |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
61 Here come the tests: | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
62 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
63 1 Some correct UTF-8 text | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
64 | |
1518 | 65 (The codepoints for this test are: | |
66 U+03BA U+1F79 U+03C3 U+03BC U+03B5 --ryan.) | | |
67 | | |
1501
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
68 You should see the Greek word 'kosme': "κόσμε" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
69 | |
1518 | 70 | |
1501
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
71 2 Boundary condition test cases | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
72 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
73 2.1 First possible sequence of a certain length | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
74 | |
1518 | 75 (byte zero skipped...there's a null added at the end of the test. --ryan.) | |
76 | | |
1501
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
77 2.1.2 2 bytes (U-00000080): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
78 2.1.3 3 bytes (U-00000800): "ࠀ" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
79 2.1.4 4 bytes (U-00010000): "𐀀" | |
1518 | 80 | |
81 (5 and 6 byte sequences were made illegal in rfc3629. --ryan.) | | |
1501
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
82 2.1.5 5 bytes (U-00200000): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
83 2.1.6 6 bytes (U-04000000): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
84 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
85 2.2 Last possible sequence of a certain length | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
86 | |
1518 | 87 2.2.1 1 byte (U-0000007F): "" | |
1501
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
88 2.2.2 2 bytes (U-000007FF): "߿" | |
1518 | 89 | |
90 (Section 5.3.2 below calls this illegal. --ryan.) | | |
1501
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
91 2.2.3 3 bytes (U-0000FFFF): "" | |
1518 | 92 | |
93 (5 and 6 bytes sequences, and 4 bytes sequences > 0x10FFFF were made illegal | | |
94 in rfc3629, so these next three should be replaced with a invalid | | |
95 character codepoint. --ryan.) | | |
1501
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
96 2.2.4 4 bytes (U-001FFFFF): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
97 2.2.5 5 bytes (U-03FFFFFF): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
98 2.2.6 6 bytes (U-7FFFFFFF): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
99 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
100 2.3 Other boundary conditions | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
101 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
102 2.3.1 U-0000D7FF = ed 9f bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
103 2.3.2 U-0000E000 = ee 80 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
104 2.3.3 U-0000FFFD = ef bf bd = "�" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
105 2.3.4 U-0010FFFF = f4 8f bf bf = "" | |
1518 | 106 | |
107 (This one is bogus in rfc3629. --ryan.) | | |
1501
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
108 2.3.5 U-00110000 = f4 90 80 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
109 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
110 3 Malformed sequences | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
111 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
112 3.1 Unexpected continuation bytes | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
113 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
114 Each unexpected continuation byte should be separately signalled as a | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
115 malformed sequence of its own. | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
116 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
117 3.1.1 First continuation byte 0x80: "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
118 3.1.2 Last continuation byte 0xbf: "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
119 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
120 3.1.3 2 continuation bytes: "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
121 3.1.4 3 continuation bytes: "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
122 3.1.5 4 continuation bytes: "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
123 3.1.6 5 continuation bytes: "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
124 3.1.7 6 continuation bytes: "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
125 3.1.8 7 continuation bytes: "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
126 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
127 3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf): | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
128 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
129 " | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
130 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
131 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
132 " | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
133 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
134 3.2 Lonely start characters | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
135 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
136 3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf), | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
137 each followed by a space character: | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
138 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
139 " | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
140 " | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
141 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
142 3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef), | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
143 each followed by a space character: | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
144 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
145 " " | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
146 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
147 3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7), | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
148 each followed by a space character: | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
149 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
150 " " | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
151 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
152 3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb), | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
153 each followed by a space character: | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
154 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
155 " " | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
156 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
157 3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd), | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
158 each followed by a space character: | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
159 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
160 " " | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
161 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
162 3.3 Sequences with last continuation byte missing | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
163 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
164 All bytes of an incomplete sequence should be signalled as a single | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
165 malformed sequence, i.e., you should see only a single replacement | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
166 character in each of the next 10 tests. (Characters as in section 2) | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
167 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
168 3.3.1 2-byte sequence with last byte missing (U+0000): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
169 3.3.2 3-byte sequence with last byte missing (U+0000): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
170 3.3.3 4-byte sequence with last byte missing (U+0000): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
171 3.3.4 5-byte sequence with last byte missing (U+0000): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
172 3.3.5 6-byte sequence with last byte missing (U+0000): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
173 3.3.6 2-byte sequence with last byte missing (U-000007FF): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
174 3.3.7 3-byte sequence with last byte missing (U-0000FFFF): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
175 3.3.8 4-byte sequence with last byte missing (U-001FFFFF): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
176 3.3.9 5-byte sequence with last byte missing (U-03FFFFFF): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
177 3.3.10 6-byte sequence with last byte missing (U-7FFFFFFF): "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
178 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
179 3.4 Concatenation of incomplete sequences | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
180 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
181 All the 10 sequences of 3.3 concatenated, you should see 10 malformed | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
182 sequences being signalled: | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
183 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
184 "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
185 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
186 3.5 Impossible bytes | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
187 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
188 The following two bytes cannot appear in a correct UTF-8 string | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
189 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
190 3.5.1 fe = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
191 3.5.2 ff = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
192 3.5.3 fe fe ff ff = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
193 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
194 4 Overlong sequences | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
195 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
196 The following sequences are not malformed according to the letter of | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
197 the Unicode 2.0 standard. However, they are longer then necessary and | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
198 a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
199 decoder" should reject them just like malformed sequences for two | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
200 reasons: (1) It helps to debug applications if overlong sequences are | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
201 not treated as valid representations of characters, because this helps | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
202 to spot problems more quickly. (2) Overlong sequences provide | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
203 alternative representations of characters, that could maliciously be | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
204 used to bypass filters that check only for ASCII characters. For | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
205 instance, a 2-byte encoded line feed (LF) would not be caught by a | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
206 line counter that counts only 0x0a bytes, but it would still be | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
207 processed as a line feed by an unsafe UTF-8 decoder later in the | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
208 pipeline. From a security point of view, ASCII compatibility of UTF-8 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
209 sequences means also, that ASCII characters are *only* allowed to be | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
210 represented by ASCII bytes in the range 0x00-0x7f. To ensure this | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
211 aspect of ASCII compatibility, use only "safe UTF-8 decoders" that | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
212 reject overlong UTF-8 sequences for which a shorter encoding exists. | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
213 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
214 4.1 Examples of an overlong ASCII character | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
215 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
216 With a safe UTF-8 decoder, all of the following five overlong | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
217 representations of the ASCII character slash ("/") should be rejected | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
218 like a malformed UTF-8 sequence, for instance by substituting it with | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
219 a replacement character. If you see a slash below, you do not have a | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
220 safe UTF-8 decoder! | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
221 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
222 4.1.1 U+002F = c0 af = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
223 4.1.2 U+002F = e0 80 af = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
224 4.1.3 U+002F = f0 80 80 af = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
225 4.1.4 U+002F = f8 80 80 80 af = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
226 4.1.5 U+002F = fc 80 80 80 80 af = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
227 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
228 4.2 Maximum overlong sequences | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
229 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
230 Below you see the highest Unicode value that is still resulting in an | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
231 overlong sequence if represented with the given number of bytes. This | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
232 is a boundary test for safe UTF-8 decoders. All five characters should | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
233 be rejected like malformed UTF-8 sequences. | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
234 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
235 4.2.1 U-0000007F = c1 bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
236 4.2.2 U-000007FF = e0 9f bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
237 4.2.3 U-0000FFFF = f0 8f bf bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
238 4.2.4 U-001FFFFF = f8 87 bf bf bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
239 4.2.5 U-03FFFFFF = fc 83 bf bf bf bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
240 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
241 4.3 Overlong representation of the NUL character | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
242 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
243 The following five sequences should also be rejected like malformed | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
244 UTF-8 sequences and should not be treated like the ASCII NUL | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
245 character. | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
246 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
247 4.3.1 U+0000 = c0 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
248 4.3.2 U+0000 = e0 80 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
249 4.3.3 U+0000 = f0 80 80 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
250 4.3.4 U+0000 = f8 80 80 80 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
251 4.3.5 U+0000 = fc 80 80 80 80 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
252 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
253 5 Illegal code positions | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
254 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
255 The following UTF-8 sequences should be rejected like malformed | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
256 sequences, because they never represent valid ISO 10646 characters and | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
257 a UTF-8 decoder that accepts them might introduce security problems | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
258 comparable to overlong UTF-8 sequences. | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
259 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
260 5.1 Single UTF-16 surrogates | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
261 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
262 5.1.1 U+D800 = ed a0 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
263 5.1.2 U+DB7F = ed ad bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
264 5.1.3 U+DB80 = ed ae 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
265 5.1.4 U+DBFF = ed af bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
266 5.1.5 U+DC00 = ed b0 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
267 5.1.6 U+DF80 = ed be 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
268 5.1.7 U+DFFF = ed bf bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
269 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
270 5.2 Paired UTF-16 surrogates | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
271 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
272 5.2.1 U+D800 U+DC00 = ed a0 80 ed b0 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
273 5.2.2 U+D800 U+DFFF = ed a0 80 ed bf bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
274 5.2.3 U+DB7F U+DC00 = ed ad bf ed b0 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
275 5.2.4 U+DB7F U+DFFF = ed ad bf ed bf bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
276 5.2.5 U+DB80 U+DC00 = ed ae 80 ed b0 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
277 5.2.6 U+DB80 U+DFFF = ed ae 80 ed bf bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
278 5.2.7 U+DBFF U+DC00 = ed af bf ed b0 80 = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
279 5.2.8 U+DBFF U+DFFF = ed af bf ed bf bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
280 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
281 5.3 Other illegal code positions | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
282 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
283 5.3.1 U+FFFE = ef bf be = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
284 5.3.2 U+FFFF = ef bf bf = "" | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
285 | |
73dc5d39bbf8
Added UTF-8 <-> UTF-16 <-> UTF-32 <-> UCS-2 <-> UCS-4 conversion capability
Sam Lantinga <slouken@libsdl.org>
parents:
diff
changeset
|
286 THE END | |
1518 | 287 |