Fixed bug #896 John Popplewell 2009-12-08 23:05:50 PST Originally reported by AKFoerster on the mailing list. Error decoding UTF8 Russian text to UTF-16LE on Windows, but specifically on platforms without iconv support (the default on Windows). Valid UTF8 characters are flagged as being overlong and then substituted by the UNKNOWN_UNICODE character. After studying the testiconv.c example program, reading the RFCs and putting some printf statements in SDL_iconv.c the problem is in a test for 'Maximum overlong sequences', specifically 4.2.1, which is carried out by the following code: } else if ( p[0] >= 0xC0 ) { if ( (p[0] & 0xE0) != 0xC0 ) { /* Skip illegal sequences return SDL_ICONV_EILSEQ; */ ch = UNKNOWN_UNICODE; } else { if ( (p[0] & 0xCE) == 0xC0 ) { <<<<<<<< here overlong = SDL_TRUE; } ch = (Uint32)(p[0] & 0x1F); left = 1; } } else { Here is the 2-byte encoding of a character in range 00000080 - 000007FF 110xxxxx 10xxxxxx The line in question is supposed to be checking for an overlong sequence which would be less than 11000001 10111111 which should be represented as a single byte. BUT, the mask value (0xCE) is wrong, it isn't checking the top-most bit: 11000001 value 11001110 mask (incorrect) ^ and should be (0xDE): 11000001 value 11011110 mask (correct) making the above code: } else if ( p[0] >= 0xC0 ) { if ( (p[0] & 0xE0) != 0xC0 ) { /* Skip illegal sequences return SDL_ICONV_EILSEQ; */ ch = UNKNOWN_UNICODE; } else { if ( (p[0] & 0xDE) == 0xC0 ) { <<<<<<<< here overlong = SDL_TRUE; } ch = (Uint32)(p[0] & 0x1F); left = 1; } } else { I can supply a test program and/or a patch if required, best regards, John Popplewell

These are test programs for the SDL library:

	checkkeys	Watch the key events to check the keyboard
	graywin		Display a gray gradient and center mouse on spacebar
	loopwave	Audio test -- loop playing a WAV file
	testalpha	Display an alpha faded icon -- paint with mouse
	testaudioinfo	Lists audio device capabilities
	testbitmap	Test displaying 1-bit bitmaps
	testblitspeed	Tests performance of SDL's blitters and converters.
	testcdrom	Sample audio CD control program
	testcursor	Tests custom mouse cursor
	testdyngl	Tests dynamically loading OpenGL library
	testerror	Tests multi-threaded error handling
	testfile	Tests RWops layer
	testgamma	Tests video device gamma ramp
	testgl		A very simple example of using OpenGL with SDL
	testgl2		An even simpler example using the SDL 1.3 API
	testhread	Hacked up test of multi-threading
	testiconv	Tests international string conversion
	testjoystick	List joysticks and watch joystick events
	testkeys	List the available keyboard keys
	testloadso	Tests the loadable library layer
	testlock	Hacked up test of multi-threading and locking
	testmultiaudio	Tests using several audio devices
	testoverlay	Tests the software/hardware overlay functionality.
	testoverlay2	Tests the overlay flickering/scaling during playback.
	testpalette	Tests palette color cycling
	testplatform	Tests types, endianness and cpu capabilities
	testsem		Tests SDL's semaphore implementation
	testsprite	Example of fast sprite movement on the screen
	testsprite2	Improved version of testsprite
	testtimer	Test the timer facilities
	testver		Check the version and dynamic loading and endianness
	testvidinfo	Show the pixel format of the display and perfom the benchmark
	testwin		Display a BMP image at various depths
	testwm		Test window manager -- title, icon, events
	testwm2		Improved version of testwm
	threadwin	Test multi-threaded event handling
	torturethread	Simple test for thread creation/destruction