Fixed bug #896 John Popplewell 2009-12-08 23:05:50 PST Originally reported by AKFoerster on the mailing list. Error decoding UTF8 Russian text to UTF-16LE on Windows, but specifically on platforms without iconv support (the default on Windows). Valid UTF8 characters are flagged as being overlong and then substituted by the UNKNOWN_UNICODE character. After studying the testiconv.c example program, reading the RFCs and putting some printf statements in SDL_iconv.c the problem is in a test for 'Maximum overlong sequences', specifically 4.2.1, which is carried out by the following code: } else if ( p[0] >= 0xC0 ) { if ( (p[0] & 0xE0) != 0xC0 ) { /* Skip illegal sequences return SDL_ICONV_EILSEQ; */ ch = UNKNOWN_UNICODE; } else { if ( (p[0] & 0xCE) == 0xC0 ) { <<<<<<<< here overlong = SDL_TRUE; } ch = (Uint32)(p[0] & 0x1F); left = 1; } } else { Here is the 2-byte encoding of a character in range 00000080 - 000007FF 110xxxxx 10xxxxxx The line in question is supposed to be checking for an overlong sequence which would be less than 11000001 10111111 which should be represented as a single byte. BUT, the mask value (0xCE) is wrong, it isn't checking the top-most bit: 11000001 value 11001110 mask (incorrect) ^ and should be (0xDE): 11000001 value 11011110 mask (correct) making the above code: } else if ( p[0] >= 0xC0 ) { if ( (p[0] & 0xE0) != 0xC0 ) { /* Skip illegal sequences return SDL_ICONV_EILSEQ; */ ch = UNKNOWN_UNICODE; } else { if ( (p[0] & 0xDE) == 0xC0 ) { <<<<<<<< here overlong = SDL_TRUE; } ch = (Uint32)(p[0] & 0x1F); left = 1; } } else { I can supply a test program and/or a patch if required, best regards, John Popplewell

   SDL Automated Testing Framework User Documentation
   by Edgar Simo Serra



   Abstract

   The SDL Automated Testing Framework, hereby after called SDL_AT, is a meant
to test the SDL code for regressions and other possible failures. It can also
be used to display what your SDL set up supports.



   Basics

   The main way to use the framework is to compile it and run it, that can be
done with the following command:

  $> make test
  
  It should then display something like:

  Platform : All tests successful (2)
  SDL_RWops : All tests successful (5)
  SDL_Surface : All tests successful (6)
  Rendering with x11 driver : All tests successful (4)

  Indicating that all tests were successful. If however a test fails output it
will report the failure to stderr indicating where and why it happened. This
output can then be sent to the developers so they can attempt to fix the
problem.



   Advanced

   By passing the "-h" or "--help" parameter to testsdl you can get an overview
of all the possible options you can set to furthur tweak the testing. A sample
of the options would be the following:

   Usage: ./testsdl [OPTIONS]
   Options are:
      -m, --manual    enables tests that require user interaction
      --noplatform    do not run the platform tests
      --norwops       do not run the rwops tests
      --nosurface     do not run the surface tests
      --norender      do not run the render tests
      -v, --verbose   increases verbosity level by 1 for each -v
      -q, --quiet     only displays errors
      -h, --help      display this message and exit


  
  Developers

  See SDL_at.h for developer information.