Mercurial > sdl-ios-xcode
view TODO @ 3539:f2846bf19360
Fixed bug #896
John Popplewell 2009-12-08 23:05:50 PST
Originally reported by AKFoerster on the mailing list.
Error decoding UTF8 Russian text to UTF-16LE on Windows, but specifically on
platforms without iconv support (the default on Windows).
Valid UTF8 characters are flagged as being overlong and then substituted by the
UNKNOWN_UNICODE character.
After studying the testiconv.c example program, reading the RFCs and putting
some printf statements in SDL_iconv.c the problem is in a test for 'Maximum
overlong sequences', specifically 4.2.1, which is carried out by the following
code:
} else if ( p[0] >= 0xC0 ) {
if ( (p[0] & 0xE0) != 0xC0 ) {
/* Skip illegal sequences
return SDL_ICONV_EILSEQ;
*/
ch = UNKNOWN_UNICODE;
} else {
if ( (p[0] & 0xCE) == 0xC0 ) { <<<<<<<< here
overlong = SDL_TRUE;
}
ch = (Uint32)(p[0] & 0x1F);
left = 1;
}
} else {
Here is the 2-byte encoding of a character in range 00000080 - 000007FF
110xxxxx 10xxxxxx
The line in question is supposed to be checking for an overlong sequence which
would be less than
11000001 10111111
which should be represented as a single byte.
BUT, the mask value (0xCE) is wrong, it isn't checking the top-most bit:
11000001 value
11001110 mask (incorrect)
^
and should be (0xDE):
11000001 value
11011110 mask (correct)
making the above code:
} else if ( p[0] >= 0xC0 ) {
if ( (p[0] & 0xE0) != 0xC0 ) {
/* Skip illegal sequences
return SDL_ICONV_EILSEQ;
*/
ch = UNKNOWN_UNICODE;
} else {
if ( (p[0] & 0xDE) == 0xC0 ) { <<<<<<<< here
overlong = SDL_TRUE;
}
ch = (Uint32)(p[0] & 0x1F);
left = 1;
}
} else {
I can supply a test program and/or a patch if required,
best regards,
John Popplewell
| author | Sam Lantinga <slouken@libsdl.org> |
|---|---|
| date | Fri, 11 Dec 2009 08:03:43 +0000 |
| parents | 15f661b279e7 |
| children | 71189305cd33 |
line wrap: on
line source
1.3 release checklist: * Create jobs@libsdl.org * Implement landscape mode for iPhone port * Add higher precision timers for profiling * Implement YUV texture support - in progress, software support is done, Ryan is working on OpenGL shaders * Implement desktop video mode change notification? * Add OpenGL 3.0 context support * Finish multiple display support * Open windows on display that is selected when creating a window * Create testing plan - becoming more and more important... - unit tests for each feature and combination with expected output and actual output comparison. * Public license for SDL_compat.c, tests, and SDL_main* * Verify mouse grab support * Properly handle mouse grab with Vista DPI scaling * Make sure the mouse is where it's supposed to be when un-grabbed * Plan how to handle events on foreign windows * Add keyboard enumeration / name query API * Implement NET_WM_PING and NET_WM_PID/WM_CLIENT_MACHINE (?) - http://standards.freedesktop.org/wm-spec/wm-spec-1.3.html#KILLINGWINDOWS * Add a timeout parameter for SDL_WaitEvent() (?) - http://lists.libsdl.org/pipermail/sdl-libsdl.org/2008-May/thread.html#65067 * Implement key repeat at the driver level * Add name -> keysym and name -> scancode lookup functions * Fix building haptic system on win32 * Add always on top window styles * Add video modes that have an alpha channel? * Implement SDL_RenderReadPixels() for screenshots and automated testing * Create extended set of (negative) error codes and have the API functions document and return them. * Add clipboard support (from scrap demo) * Add support for desktop windows? * Add support for synchronizing 2D updates with vblank (bug #406) * Add support for real-time threads and/or adjusting thread priority? * Add support for querying the number of CPUs and binding threads to them? * Allow setting multiple icon sizes Wish list for the 1.3 development branch: http://bugzilla.libsdl.org/ * Add a way to register custom events * DirectInput joystick support needs to be implemented * Explicit vertical retrace wait (maybe separate from SDL_Flip?) * Shaped windows, windows without borders * Multiple display support * Add a timestamp to events * Add audio input API * Add modifier state to key and button events In the jump from 1.2 to 1.3, we should change the SDL_Rect members to int and evaluate all the rest of the datatypes. This is the only place we should do it though, since the 1.2 series should not break binary compatibility in this way. Requests: * PCM volume control (deprecated, but possible)