Mercurial > sdl-ios-xcode
view test/utf8.txt @ 5067:61d53410eb41
Fixed bug #859
CREATE_SUBDIRS helps a lot if browsing HTML documentation in a file browser.
ALWAYS_DETAILED_SEC makes sure everything has at least the automatic
documentation like function prototype and source references.
STRIP_FROM_PATH allows you to include only the relevant portions of the files'
paths, cleaning up both the file list and directory tree, though you need to
change the path listed here to match wherever you put SDL.
ALIASES avoids some warnings generated by
C:\source\svn.libsdl.org\trunk\SDL\src\joystick\darwin\10.3.9-FIX\IOHIDLib.h.
It seems Apple uses a few commands which are not normally supported by Doxygen.
BUILTIN_STL_SUPPORT adds support for parsing code which makes use of the
standard template library. There isn't a lot of C++ in SDL (some in bwindow at
least), but this still seems like a good idea.
TYPEDEF_HIDES_STRUCT means that for code like this:
typedef struct A {int B;} C;
C is documented as a structure containing B instead of a typedef mapped to A.
EXTRACT_ALL, EXTRACT_PRIVATE, EXTRACT_STATIC, EXTRACT_LOCAL_METHODS,
EXTRACT_ANON_NSPACES and INTERNAL_DOCS make sure that _everything_ is
documented.
CASE_SENSE_NAMES = NO avoids potential conflicts when building documentation on
case insensitive file systems like NTFS and FAT32.
WARN_NO_PARAMDOC lets you know when you have documented some, but not all, of
the parameters of a function. This is useful when you're working on adding
such documentation since it makes partially documented functions easier to
spot.
WARN_LOGFILE writes warnings to a seperate file instead of mixing them in with
stdout. When not running in quiet mode, these warnings can be hard to spot
without this flag.
I added *.h.in and *.h.default to FILE_PATTERNS to generate documentation for
config.h.in and config.h.default.
RECURSIVE tells doxygen to look not only in the input directory, but also in
subfolders.
EXCLUDE avoids documenting things like test programs, examples and templates
which need to be documented separately.
I've used EXCLUDE_PATTERNS to exclude non-source subdirectories that often find
their way into source folders (such as obj or .svn).
EXAMPLE_PATH lists directories doxygen will search to find included example
code. So far, SDL doesn't really use this feature, but I've listed some likely
locations.
SOURCE_BROWSER adds syntax highlighted source code to the HTML output.
USE_HTAGS is nice, but not available on Windows.
INLINE_SOURCES adds the body of a function to it's documentation so you can
quickly see exactly what it does.
ALPHABETICAL_INDEX generates an alphabetical list of all structures, functions,
etc., which makes it much easier to find what you're looking for.
IGNORE_PREFIX skips the SDL_ prefix when deciding which index page to place an
item on so you don't have everything show up under "S".
HTML_DYNAMIC_SECTIONS hides the includes/included by diagrams by default and
adds JavaScript to allow the user to show and hide them by clicking a link.
ENUM_VALUES_PER_LINE = 1 makes enums easier to read by placing each value on
it's own line.
GENERATE_TREEVIEW produces a two frame index page with a navigation tree on the
left.
I have LaTeX and man pages turned off to speed up doxygen, you may want to turn
them back on yourself.
I added _WIN32=1 to PREDEFINED to cause SDL to output documentation related to
Win32 builds of SDL. Normally, doxygen gets confused since there are multiple
definitions for various structures and formats that vary by platform. Without
this doxygen can produce broken documentation or, if you're lucky, output
documentation only for the dummy drivers, which isn't very useful. You need to
pick a platform.
GENERATE_TAGFILE produces a file which can be used to link other doxygen
documentation to the SDL documentation.
CLASS_DIAGRAMS turns on class diagrams even when dot is not available.
HAVE_DOT tells doxygen to try to use dot to generate diagrams.
TEMPLATE_RELATIONS and INCLUDE_GRAPH add additional diagrams to the
documentation.
DOT_MULTI_TARGETS speeds up dot.
OUTPUT_DIRECTORY, INPUT and other paths reflect the fact that this Doxyfile is
intended to process src as well as include and is being run from a separate
subdirectory. Doxygen produces several temporary files while it's running and
if interrupted, can leave those files behind. It's easier to clean up if there
aren't a hundred or so files in the same folder. I typically run doxygen in
SDL/doxy and set the output directory to '.'. Since doxygen puts it's output
in subfolders by type, this keeps things pretty well organised. You could use
'../doc' instead and get the same results.
| author | Sam Lantinga <slouken@libsdl.org> |
|---|---|
| date | Fri, 21 Jan 2011 12:57:01 -0800 |
| parents | 4d711949cd9a |
| children |
line wrap: on
line source
UTF-8 decoder capability and stress test ---------------------------------------- Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2003-02-19 This test file can help you examine, how your UTF-8 decoder handles various types of correct, malformed, or otherwise interesting UTF-8 sequences. This file is not meant to be a conformance test. It does not prescribes any particular outcome and therefore there is no way to "pass" or "fail" this test file, even though the texts suggests a preferable decoder behaviour at some places. The aim is instead to help you think about and test the behaviour of your UTF-8 on a systematic collection of unusual inputs. Experience so far suggests that most first-time authors of UTF-8 decoders find at least one serious problem in their decoder by using this file. The test lines below cover boundary conditions, malformed UTF-8 sequences as well as correctly encoded UTF-8 sequences of Unicode code points that should never occur in a correct UTF-8 file. According to ISO 10646-1:2000, sections D.7 and 2.3c, a device receiving UTF-8 shall interpret a "malformed sequence in the same way that it interprets a character that is outside the adopted subset" and "characters that are not within the adopted subset shall be indicated to the user" by a receiving device. A quite commonly used approach in UTF-8 decoders is to replace any malformed UTF-8 sequence by a replacement character (U+FFFD), which looks a bit like an inverted question mark, or a similar symbol. It might be a good idea to visually distinguish a malformed UTF-8 sequence from a correctly encoded Unicode character that is just not available in the current font but otherwise fully legal, even though ISO 10646-1 doesn't mandate this. In any case, just ignoring malformed sequences or unavailable characters does not conform to ISO 10646, will make debugging more difficult, and can lead to user confusion. Please check, whether a malformed UTF-8 sequence is (1) represented at all, (2) represented by exactly one single replacement character (or equivalent signal), and (3) the following quotation mark after an illegal UTF-8 sequence is correctly displayed, i.e. proper resynchronization takes place immageately after any malformed sequence. This file says "THE END" in the last line, so if you don't see that, your decoder crashed somehow before, which should always be cause for concern. All lines in this file are exactly 79 characters long (plus the line feed). In addition, all lines end with "|", except for the two test lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls U+0000 and U+007F. If you display this file with a fixed-width font, these "|" characters should all line up in column 79 (right margin). This allows you to test quickly, whether your UTF-8 decoder finds the correct number of characters in every line, that is whether each malformed sequences is replaced by a single replacement character. Note that as an alternative to the notion of malformed sequence used here, it is also a perfectly acceptable (and in some situations even preferable) solution to represent each individual byte of a malformed sequence by a replacement character. If you follow this strategy in your decoder, then please ignore the "|" column. Here come the tests: | | 1 Some correct UTF-8 text | | (The codepoints for this test are: | U+03BA U+1F79 U+03C3 U+03BC U+03B5 --ryan.) | | You should see the Greek word 'kosme': "κόσμε" | | | 2 Boundary condition test cases | | 2.1 First possible sequence of a certain length | | (byte zero skipped...there's a null added at the end of the test. --ryan.) | | 2.1.2 2 bytes (U-00000080): "" | 2.1.3 3 bytes (U-00000800): "ࠀ" | 2.1.4 4 bytes (U-00010000): "𐀀" | | (5 and 6 byte sequences were made illegal in rfc3629. --ryan.) | 2.1.5 5 bytes (U-00200000): "" | 2.1.6 6 bytes (U-04000000): "" | | 2.2 Last possible sequence of a certain length | | 2.2.1 1 byte (U-0000007F): "" | 2.2.2 2 bytes (U-000007FF): "߿" | | (Section 5.3.2 below calls this illegal. --ryan.) | 2.2.3 3 bytes (U-0000FFFF): "" | | (5 and 6 bytes sequences, and 4 bytes sequences > 0x10FFFF were made illegal | in rfc3629, so these next three should be replaced with a invalid | character codepoint. --ryan.) | 2.2.4 4 bytes (U-001FFFFF): "" | 2.2.5 5 bytes (U-03FFFFFF): "" | 2.2.6 6 bytes (U-7FFFFFFF): "" | | 2.3 Other boundary conditions | | 2.3.1 U-0000D7FF = ed 9f bf = "" | 2.3.2 U-0000E000 = ee 80 80 = "" | 2.3.3 U-0000FFFD = ef bf bd = "�" | 2.3.4 U-0010FFFF = f4 8f bf bf = "" | | (This one is bogus in rfc3629. --ryan.) | 2.3.5 U-00110000 = f4 90 80 80 = "" | | 3 Malformed sequences | | 3.1 Unexpected continuation bytes | | Each unexpected continuation byte should be separately signalled as a | malformed sequence of its own. | | 3.1.1 First continuation byte 0x80: "" | 3.1.2 Last continuation byte 0xbf: "" | | 3.1.3 2 continuation bytes: "" | 3.1.4 3 continuation bytes: "" | 3.1.5 4 continuation bytes: "" | 3.1.6 5 continuation bytes: "" | 3.1.7 6 continuation bytes: "" | 3.1.8 7 continuation bytes: "" | | 3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf): | | " | | | " | | 3.2 Lonely start characters | | 3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf), | each followed by a space character: | | " | " | | 3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef), | each followed by a space character: | | " " | | 3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7), | each followed by a space character: | | " " | | 3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb), | each followed by a space character: | | " " | | 3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd), | each followed by a space character: | | " " | | 3.3 Sequences with last continuation byte missing | | All bytes of an incomplete sequence should be signalled as a single | malformed sequence, i.e., you should see only a single replacement | character in each of the next 10 tests. (Characters as in section 2) | | 3.3.1 2-byte sequence with last byte missing (U+0000): "" | 3.3.2 3-byte sequence with last byte missing (U+0000): "" | 3.3.3 4-byte sequence with last byte missing (U+0000): "" | 3.3.4 5-byte sequence with last byte missing (U+0000): "" | 3.3.5 6-byte sequence with last byte missing (U+0000): "" | 3.3.6 2-byte sequence with last byte missing (U-000007FF): "" | 3.3.7 3-byte sequence with last byte missing (U-0000FFFF): "" | 3.3.8 4-byte sequence with last byte missing (U-001FFFFF): "" | 3.3.9 5-byte sequence with last byte missing (U-03FFFFFF): "" | 3.3.10 6-byte sequence with last byte missing (U-7FFFFFFF): "" | | 3.4 Concatenation of incomplete sequences | | All the 10 sequences of 3.3 concatenated, you should see 10 malformed | sequences being signalled: | | "" | | 3.5 Impossible bytes | | The following two bytes cannot appear in a correct UTF-8 string | | 3.5.1 fe = "" | 3.5.2 ff = "" | 3.5.3 fe fe ff ff = "" | | 4 Overlong sequences | | The following sequences are not malformed according to the letter of | the Unicode 2.0 standard. However, they are longer then necessary and | a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8 | decoder" should reject them just like malformed sequences for two | reasons: (1) It helps to debug applications if overlong sequences are | not treated as valid representations of characters, because this helps | to spot problems more quickly. (2) Overlong sequences provide | alternative representations of characters, that could maliciously be | used to bypass filters that check only for ASCII characters. For | instance, a 2-byte encoded line feed (LF) would not be caught by a | line counter that counts only 0x0a bytes, but it would still be | processed as a line feed by an unsafe UTF-8 decoder later in the | pipeline. From a security point of view, ASCII compatibility of UTF-8 | sequences means also, that ASCII characters are *only* allowed to be | represented by ASCII bytes in the range 0x00-0x7f. To ensure this | aspect of ASCII compatibility, use only "safe UTF-8 decoders" that | reject overlong UTF-8 sequences for which a shorter encoding exists. | | 4.1 Examples of an overlong ASCII character | | With a safe UTF-8 decoder, all of the following five overlong | representations of the ASCII character slash ("/") should be rejected | like a malformed UTF-8 sequence, for instance by substituting it with | a replacement character. If you see a slash below, you do not have a | safe UTF-8 decoder! | | 4.1.1 U+002F = c0 af = "" | 4.1.2 U+002F = e0 80 af = "" | 4.1.3 U+002F = f0 80 80 af = "" | 4.1.4 U+002F = f8 80 80 80 af = "" | 4.1.5 U+002F = fc 80 80 80 80 af = "" | | 4.2 Maximum overlong sequences | | Below you see the highest Unicode value that is still resulting in an | overlong sequence if represented with the given number of bytes. This | is a boundary test for safe UTF-8 decoders. All five characters should | be rejected like malformed UTF-8 sequences. | | 4.2.1 U-0000007F = c1 bf = "" | 4.2.2 U-000007FF = e0 9f bf = "" | 4.2.3 U-0000FFFF = f0 8f bf bf = "" | 4.2.4 U-001FFFFF = f8 87 bf bf bf = "" | 4.2.5 U-03FFFFFF = fc 83 bf bf bf bf = "" | | 4.3 Overlong representation of the NUL character | | The following five sequences should also be rejected like malformed | UTF-8 sequences and should not be treated like the ASCII NUL | character. | | 4.3.1 U+0000 = c0 80 = "" | 4.3.2 U+0000 = e0 80 80 = "" | 4.3.3 U+0000 = f0 80 80 80 = "" | 4.3.4 U+0000 = f8 80 80 80 80 = "" | 4.3.5 U+0000 = fc 80 80 80 80 80 = "" | | 5 Illegal code positions | | The following UTF-8 sequences should be rejected like malformed | sequences, because they never represent valid ISO 10646 characters and | a UTF-8 decoder that accepts them might introduce security problems | comparable to overlong UTF-8 sequences. | | 5.1 Single UTF-16 surrogates | | 5.1.1 U+D800 = ed a0 80 = "" | 5.1.2 U+DB7F = ed ad bf = "" | 5.1.3 U+DB80 = ed ae 80 = "" | 5.1.4 U+DBFF = ed af bf = "" | 5.1.5 U+DC00 = ed b0 80 = "" | 5.1.6 U+DF80 = ed be 80 = "" | 5.1.7 U+DFFF = ed bf bf = "" | | 5.2 Paired UTF-16 surrogates | | 5.2.1 U+D800 U+DC00 = ed a0 80 ed b0 80 = "" | 5.2.2 U+D800 U+DFFF = ed a0 80 ed bf bf = "" | 5.2.3 U+DB7F U+DC00 = ed ad bf ed b0 80 = "" | 5.2.4 U+DB7F U+DFFF = ed ad bf ed bf bf = "" | 5.2.5 U+DB80 U+DC00 = ed ae 80 ed b0 80 = "" | 5.2.6 U+DB80 U+DFFF = ed ae 80 ed bf bf = "" | 5.2.7 U+DBFF U+DC00 = ed af bf ed b0 80 = "" | 5.2.8 U+DBFF U+DFFF = ed af bf ed bf bf = "" | | 5.3 Other illegal code positions | | 5.3.1 U+FFFE = ef bf be = "" | 5.3.2 U+FFFF = ef bf bf = "" | | THE END |