Mercurial > lcfOS

comparison python/c3/lexer.py @ 213:003c8a976fff

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Merge of semantics and parser again ..
author Windel Bouwman
date Fri, 05 Jul 2013 11:18:48 +0200
parents d77cb5962cc5
children 848c4b15fd0b
comparison
equal deleted inserted replaced
212:62386bcee1ba 213:003c8a976fff
4 4
5 """ 5 """
6 Lexical analyzer part. Splits the input character stream into tokens. 6 Lexical analyzer part. Splits the input character stream into tokens.
7 """ 7 """
8 8
9 keywords = ['and', 'or', 'not','true', 'false', \ 9 keywords = ['and', 'or', 'not', 'true', 'false', \
10 'else', 'if', 'while', 'return', \ 10 'else', 'if', 'while', 'return', \
11 'function', 'var', 'type', 'const', \ 11 'function', 'var', 'type', 'const', \
12 'struct', \
12 'import', 'package' ] 13 'import', 'package' ]
13 14
14 def tokenize(s): 15 def tokenize(s):
15 """ 16 """
16 Tokenizer, generates an iterator that 17 Tokenizer, generates an iterator that