# HG changeset patch
# User Sam Lantinga <slouken@libsdl.org>
# Date 1260518457 0
# Node ID 1524d3237820c1c102d773f5061280266e0ed73d
# Parent  72d3b4fd918b661a06e237b1b9e2dcb2b72e9ed4
Fixed bug #896

 John Popplewell      2009-12-08 23:05:50 PST

Originally reported by AKFoerster on the mailing list.

Error decoding UTF8 Russian text to UTF-16LE on Windows, but specifically on
platforms without iconv support (the default on Windows).

Valid UTF8 characters are flagged as being overlong and then substituted by the
UNKNOWN_UNICODE character.

After studying the testiconv.c example program, reading the RFCs and putting
some printf statements in SDL_iconv.c the problem is in a test for 'Maximum
overlong sequences', specifically 4.2.1, which is carried out by the following
code:

      } else if ( p[0] >= 0xC0 ) {
        if ( (p[0] & 0xE0) != 0xC0 ) {
          /* Skip illegal sequences
            return SDL_ICONV_EILSEQ;
          */
          ch = UNKNOWN_UNICODE;
        } else {
          if ( (p[0] & 0xCE) == 0xC0 ) {    <<<<<<<< here
            overlong = SDL_TRUE;
          }
          ch = (Uint32)(p[0] & 0x1F);
          left = 1;
        }
      } else {

Here is the 2-byte encoding of a character in range 00000080 - 000007FF
    110xxxxx 10xxxxxx

The line in question is supposed to be checking for an overlong sequence which
would be less than
    11000001 10111111

which should be represented as a single byte.

BUT, the mask value (0xCE) is wrong, it isn't checking the top-most bit:
    11000001     value
    11001110     mask (incorrect)
       ^
and should be (0xDE):
    11000001     value
    11011110     mask (correct)

making the above code:

      } else if ( p[0] >= 0xC0 ) {
        if ( (p[0] & 0xE0) != 0xC0 ) {
          /* Skip illegal sequences
            return SDL_ICONV_EILSEQ;
          */
          ch = UNKNOWN_UNICODE;
        } else {
          if ( (p[0] & 0xDE) == 0xC0 ) {    <<<<<<<< here
            overlong = SDL_TRUE;
          }
          ch = (Uint32)(p[0] & 0x1F);
          left = 1;
        }
      } else {

I can supply a test program and/or a patch if required,

best regards,
John Popplewell

diff -r 72d3b4fd918b -r 1524d3237820 src/stdlib/SDL_iconv.c
--- a/src/stdlib/SDL_iconv.c	Thu Dec 03 04:20:46 2009 +0000
+++ b/src/stdlib/SDL_iconv.c	Fri Dec 11 08:00:57 2009 +0000
@@ -383,7 +383,7 @@
 						*/
 						ch = UNKNOWN_UNICODE;
 					} else {
-						if ( (p[0] & 0xCE) == 0xC0 ) {
+						if ( (p[0] & 0xDE) == 0xC0 ) {
 							overlong = SDL_TRUE;
 						}
 						ch = (Uint32)(p[0] & 0x1F);