Fixed bug #896 John Popplewell 2009-12-08 23:05:50 PST Originally reported by AKFoerster on the mailing list. Error decoding UTF8 Russian text to UTF-16LE on Windows, but specifically on platforms without iconv support (the default on Windows). Valid UTF8 characters are flagged as being overlong and then substituted by the UNKNOWN_UNICODE character. After studying the testiconv.c example program, reading the RFCs and putting some printf statements in SDL_iconv.c the problem is in a test for 'Maximum overlong sequences', specifically 4.2.1, which is carried out by the following code: } else if ( p[0] >= 0xC0 ) { if ( (p[0] & 0xE0) != 0xC0 ) { /* Skip illegal sequences return SDL_ICONV_EILSEQ; */ ch = UNKNOWN_UNICODE; } else { if ( (p[0] & 0xCE) == 0xC0 ) { <<<<<<<< here overlong = SDL_TRUE; } ch = (Uint32)(p[0] & 0x1F); left = 1; } } else { Here is the 2-byte encoding of a character in range 00000080 - 000007FF 110xxxxx 10xxxxxx The line in question is supposed to be checking for an overlong sequence which would be less than 11000001 10111111 which should be represented as a single byte. BUT, the mask value (0xCE) is wrong, it isn't checking the top-most bit: 11000001 value 11001110 mask (incorrect) ^ and should be (0xDE): 11000001 value 11011110 mask (correct) making the above code: } else if ( p[0] >= 0xC0 ) { if ( (p[0] & 0xE0) != 0xC0 ) { /* Skip illegal sequences return SDL_ICONV_EILSEQ; */ ch = UNKNOWN_UNICODE; } else { if ( (p[0] & 0xDE) == 0xC0 ) { <<<<<<<< here overlong = SDL_TRUE; } ch = (Uint32)(p[0] & 0x1F); left = 1; } } else { I can supply a test program and/or a patch if required, best regards, John Popplewell

Wish list for the 1.3 development branch:
http://bugzilla.libsdl.org/

 * Add mousewheel events (new unified event architecture?)
 * DirectInput joystick support needs to be implemented
 * Be able to enumerate and select available audio and video drivers
 * Fullscreen video mode support for Mac OS X
 * Explicit vertical retrace wait (maybe separate from SDL_Flip?)
 * Shaped windows, windows without borders
 * Multiple windows, multiple display support
 * SDL_INIT_EVENTTHREAD on Windows and MacOS?
 * Add a timestamp to events
 * Add audio input API
 * Add hardware accelerated scaled blit
 * Add hardware accelerated alpha blits
 * Redesign blitting architecture to allow blit plugins

In the jump from 1.2 to 1.3, we should change the SDL_Rect members to
int and evaluate all the rest of the datatypes.  This is the only place
we should do it though, since the 1.2 series should not break binary
compatibility in this way.

Requests:
 * PCM and CDROM volume control (deprecated, but possible)