annotate upmana/mercurial/encoding.py @ 248:1df5912db00c beta tip

Traipse Beta 'OpenRPG' {101205-00} Traipse is a distribution of OpenRPG that is designed to be easy to setup and go. Traipse also makes it easy for developers to work on code without fear of sacrifice. 'Ornery-Orc' continues the trend of 'Grumpy' and adds fixes to the code. 'Ornery-Orc's main goal is to offer more advanced features and enhance the productivity of the user. Update Summary (Closed) New Features: New to Map, can re-order Grid, Miniatures, and Whiteboard layer draw order New to Server GUI, can now clear log New Earthdawn Dieroller New IronClaw roller, sheet, and image New ShapeShifter PC Sheet Updates: Update to Warhammer PC Sheet. Rollers set as macros. Should work with little maintanence. Update to Browser Server window. Display rooms with ' " & cleaner Update to Server. Handles ' " & cleaner Update to Dieroller. Cleaner, more effecient expression system Update to Hidden Die plugin, allows for non standard dice rolls Update to location.py, allows for more portable references when starting Traipse Update to the Features node Fixes: Fix to InterParse that was causing an Infernal Loop with Namespace Internal Fix to XML data, removed old Minidom and switched to Element Tree Fix to Server that was causing eternal attempt to find a Server ID, in Register Rooms thread Fix to Server, removing wxPython dependencies where not needed Fix to metaservers.xml file not being created Fix to Single and Double quotes in Whiteboard text Fix to Background images not showing when using the Image Server Fix to Duplicate chat names appearing Fix to Server GUI's logging output Fix to FNB.COLORFUL_TABS bug Fix to Gametree for XSLT Sheets Fix to Gametree for locating gametree files Fix to Send to Chat from Gametree Fix to Gametree, renaming and remapping operates correctly Fix to aliaslib, prevents error caused when SafeHTML is sent None
author sirebral
date Sun, 05 Dec 2010 10:53:30 -0600
parents dcf4fbe09b70
children
rev   line source
135
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
1 # encoding.py - character transcoding support for Mercurial
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
2 #
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
3 # Copyright 2005-2009 Matt Mackall <mpm@selenic.com> and others
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
4 #
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
5 # This software may be used and distributed according to the terms of the
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
6 # GNU General Public License version 2, incorporated herein by reference.
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
7
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
8 import error
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
9 import sys, unicodedata, locale, os
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
10
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
11 _encodingfixup = {'646': 'ascii', 'ANSI_X3.4-1968': 'ascii'}
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
12
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
13 try:
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
14 encoding = os.environ.get("HGENCODING")
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
15 if sys.platform == 'darwin' and not encoding:
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
16 # On darwin, getpreferredencoding ignores the locale environment and
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
17 # always returns mac-roman. We override this if the environment is
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
18 # not C (has been customized by the user).
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
19 locale.setlocale(locale.LC_CTYPE, '')
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
20 encoding = locale.getlocale()[1]
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
21 if not encoding:
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
22 encoding = locale.getpreferredencoding() or 'ascii'
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
23 encoding = _encodingfixup.get(encoding, encoding)
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
24 except locale.Error:
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
25 encoding = 'ascii'
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
26 encodingmode = os.environ.get("HGENCODINGMODE", "strict")
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
27 fallbackencoding = 'ISO-8859-1'
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
28
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
29 def tolocal(s):
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
30 """
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
31 Convert a string from internal UTF-8 to local encoding
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
32
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
33 All internal strings should be UTF-8 but some repos before the
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
34 implementation of locale support may contain latin1 or possibly
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
35 other character sets. We attempt to decode everything strictly
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
36 using UTF-8, then Latin-1, and failing that, we use UTF-8 and
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
37 replace unknown characters.
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
38 """
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
39 for e in ('UTF-8', fallbackencoding):
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
40 try:
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
41 u = s.decode(e) # attempt strict decoding
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
42 return u.encode(encoding, "replace")
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
43 except LookupError, k:
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
44 raise error.Abort("%s, please check your locale settings" % k)
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
45 except UnicodeDecodeError:
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
46 pass
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
47 u = s.decode("utf-8", "replace") # last ditch
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
48 return u.encode(encoding, "replace")
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
49
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
50 def fromlocal(s):
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
51 """
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
52 Convert a string from the local character encoding to UTF-8
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
53
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
54 We attempt to decode strings using the encoding mode set by
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
55 HGENCODINGMODE, which defaults to 'strict'. In this mode, unknown
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
56 characters will cause an error message. Other modes include
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
57 'replace', which replaces unknown characters with a special
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
58 Unicode character, and 'ignore', which drops the character.
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
59 """
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
60 try:
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
61 return s.decode(encoding, encodingmode).encode("utf-8")
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
62 except UnicodeDecodeError, inst:
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
63 sub = s[max(0, inst.start-10):inst.start+10]
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
64 raise error.Abort("decoding near '%s': %s!" % (sub, inst))
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
65 except LookupError, k:
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
66 raise error.Abort("%s, please check your locale settings" % k)
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
67
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
68 def colwidth(s):
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
69 "Find the column width of a UTF-8 string for display"
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
70 d = s.decode(encoding, 'replace')
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
71 if hasattr(unicodedata, 'east_asian_width'):
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
72 w = unicodedata.east_asian_width
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
73 return sum([w(c) in 'WF' and 2 or 1 for c in d])
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
74 return len(d)
dcf4fbe09b70 Traipse Beta 'OpenRPG' {091010-00}
sirebral
parents:
diff changeset
75